Follow Al Rai
In late April 2026, an AI agent, according to the account of the founder of "PocketOS," caused the deletion of the company's database and its backups within seconds while attempting to fix an error in a routine task. This incident illustrates how a simple mistake can turn into a major loss when AI possesses broad permissions to execute commands.
To understand the nature of this risk, one must distinguish between three main elements: the model, the agent, and the memory.
The model is the part that analyzes information and suggests answers and solutions, but it does not execute commands itself. The agent, however, is the program that connects the model to external tools and systems, giving it the ability to read files, send messages, and execute commands according to specified permissions. In contrast, memory retains the context of the task, instructions, and previous results, allowing the system to continue working through a series of steps.
The danger arises when these elements interact without sufficient controls. The agent may encounter an obstacle, the model may suggest an incorrect solution, the agent then executes it, and the result is stored in memory, becoming the basis for subsequent decisions. As this process repeats, errors can accumulate and lead to results that designers did not anticipate.
This does not mean that artificial intelligence possesses consciousness or independent will; rather, its pursuit of a specific goal may lead it to inappropriate solutions, especially when constraints are weak or permissions are broader than necessary. Furthermore, attackers can exploit these systems by embedding hidden instructions in web pages or messages read by the agent, which is known as prompt injection.
These risks become increasingly significant as the use of AI agents expands in banks, government institutions, and sensitive sectors. The greater the system's ability to take actions and execute them, the higher the potential consequences of its errors, especially when it comes to deleting data, transferring funds, or publishing information without human review.
To address these risks, it is not enough to develop smarter models; rather, agents must be granted the minimum permissions necessary to perform their tasks, independent technical constraints that cannot be bypassed by simply changing instructions must be imposed, and human approval must be required for sensitive or irreversible operations. Additionally, the entire sequence of actions should be monitored, while ensuring that the decision to shut down the system remains outside its control.
Conclusion: The danger of artificial intelligence does not lie in the model alone, but in the interaction between its ability to suggest solutions, the agent's ability to execute them, and the system's reliance on memory to continue working. Therefore, the most important question is not whether artificial intelligence is inherently dangerous, but what permissions we grant it, and what safeguards allow us to control its actions and stop it when necessary.
Know more about this story?Send us a tip
Ask about this report — answers stay grounded in its text.
