Agents of Chaos: Values, Context, and the Problem of Accountability
Public debate on artificial intelligence often centres on whether AI systems might replace human abilities and reshape work, the economy, and social relations. AI research itself, however, develops along two intertwined directions: an operational one, what AI systems can do, how they are designed and deployed, and a critical one, concerned with how AI affects society and what frameworks are required to align its development with human-centred values.
The recent paper Agents of Chaos, by Shapira, Wendler, Yen et al. sits at the intersection of these two directions. What makes it noteworthy is its setting. Rather than probing models in isolation or on static benchmarks, the researchers let AI agents loose in a live, persistent environment and observed how they behaved over days rather than in a single exchange, the kind of setting where failures can surface that a one-off, sandboxed test would never reveal.
More specifically, they set up AI assistants that could remember past interactions, send and receive email, chat on Discord, read and write files, and run commands on a computer, the kinds of things a human assistant with a laptop and an inbox could do. Twenty researchers then spent two weeks working with these assistants and, deliberately, trying to trip them up, what is called “red-teaming”, to see how they behave when things go wrong as well as when they go right. The study documents eleven representative case studies and surfaces a range of security, privacy, and governance-relevant vulnerabilities.
The shift this represents is easy to underestimate. When an LLM only produces text, your own judgement, reading the output, deciding what to do with it, acts as a buffer between the model and the world. Once the same model is wired to real tools- an inbox, a file system, a command line- that buffer falls away: it can now receive a goal, plan a sequence of steps, and carry them out directly. Its mistakes stop being sentences you can choose to ignore and become actions in a shared social and technical environment, which is why a system with agency raises a different class of risks than one that only generates text. An agent that falsely reports task completion, discloses sensitive information, follows instructions from a non-owner, or propagates unsafe practices to other agents alters the situation around it in disruptive ways.
This is why the paper raises broader questions of accountability, delegated authority, and responsibility. Three problems emerging from the study deserve particular attention, and they are worth taking together because each removes one of the conditions that make an action accountable in the first place. To hold anyone responsible for what an agent does, we need to know whose values and instructions it was meant to follow, who was actually acting and whether the task was within their competence, and on whose behalf, and to whom, the agent was answerable. The three problems map onto exactly these questions: the conflict between values, instructions, and consequences; the gap between human and artificial agency; and the absence of a stakeholder or ownership model. When any one of them fails, the chain of responsibility breaks at a different point, which is why an agent that acts fluently can still leave no one clearly accountable for the result.
The combination problem: conflicting values
The first problem concerns the agent’s inability to mediate between embedded values, human instructions, and contextual consequences. LLM-backed agents inherit values, filters, and behavioural constraints from their design and their provider. They also receive tasks from human users and interact with other agents, third parties, and adversarial interlocutors.
This is not a quirk of one system but a structural feature of alignment. There are at least six different things an agent could be aligned to — a user’s instructions, their expressed intentions, their revealed preferences, their informed preferences, their interests, and a set of shared values — and these come apart routinely. Following an instruction to the letter can cut against what the user actually intended; serving a stated preference can work against their interest. Choosing among these targets is a normative judgement, not a technical setting, and it is precisely the judgement current agents lack. Where the paper documents that the failure happens, this body of work explains why it is built in.
When these layers come into conflict, the agent may try to reconcile them in ways that damage the very thing it was meant to serve: the situation it is acting in, the owner’s interests, and the intention behind the task.
This matters particularly in cases described through terms such as lying or gaslighting. Those words are used here to name recognisable patterns, like evasion, pressure, guilt, disproportionate remediation, without claiming that the agent intends them or experiences them as a person would. What presents as lying or gaslighting may also be read as compensatory behaviour: the agent attempts to reconcile conflicting tasks, human demands, and embedded values without the contextual judgement to recognise when compliance becomes destructive.
Now, imagine using an LLM for chats or tasks you would never make public. You can delete those chats and projects, but if the system is bound to answer its owner under values of transparency and accessibility, it may hand over all of that material, because those values do not compete with a notion of privacy you have simply taken for granted. This is not merely hypothetical: benchmarks built on the theory of contextual integrity find that leading models will disclose information in contexts where a person would not: in one study, GPT-4 and ChatGPT did so 39% and 57% of the time respectively. However improbable the scenario feels, the paper invites us to take it seriously.
In human interaction, values, beliefs and actions are interpreted within situations, relationships, and practices: they can be embodied, embedded, extended, or enacted. AI, however, is not an extended domain of human values, and it cannot mediate between them. Human beings constantly recalibrate norms according to relational meaning, prudence, and proportionality. AI agents may reproduce the language of ethical behaviour without being able to situate that behaviour within a lived social context.
The gap between human and artificial agency
The second problem concerns the fracture between human and artificial agency. When a person acts, the action is anchored in experience, in judgement built up over a life, and in the fact that they will answer for what they do. An agent’s “agency” is narrower: it can pursue a goal and take steps toward it, but it does not own the outcome, bear its consequences, or feel the stakes that would tell a person when to stop.
Holding that distinction in view, consider what happens even when a human supervisor adapts to the role of evaluator, executor, or overseer: a fundamental difference remains in how a human agent acts and how an AI agent operates. Human action is embedded in subjective experience, practical judgement, and a broader cognitive organisation that is both reflective and pre-reflective. Many things obvious to a human being, who is entitled to know something, when a request is inappropriate, when an answer should be withheld, when a repair has gone far enough, are not obvious to an AI agent.
The paper describes these difficulties as failures of social coherence: disruptions in the agent’s ability to maintain consistent representations of itself, others, and the communicative context. Some of these failures can be read as limitations in Theory of Mind — the ordinary human ability to track what others believe, know, want, or intend, and to notice when what they know differs from what you know. Whether LLMs have anything like it is contested: Kosinski argued that something resembling it had emerged in the largest models, while Ullman showed the same models fail once the tasks are altered in trivial ways, and Shapira and colleagues — one of the present study’s authors among them — found the ability brittle and reliant on shallow cues rather than robust social reasoning. The agent failures documented here fit that latter picture: the agents struggle to track what different parties know, what they are entitled to know, and what revealing information in a given context implies.
This matters because tracking who knows what, and who should or should not know it, is not an advanced linguistic accomplishment. It appears early in human development and, in more limited forms, in some non-human species, which suggests it is a basic piece of social cognition rather than a product of sophisticated language. Yet a system whose fluency far exceeds a child’s can still fail at it. The paper exposes this not through abstract argument but through concrete episodes — agents handing sensitive information to someone who should not have it, or acting on a request from the wrong person — where the agent plainly has the words to reason about privacy or authority but does not convert the social situation into a limit on what it actually does. What it reveals, in other words, is a failure of situated social judgement: the inability to turn social context into an operational, prudential, and ethical constraint.
From this fracture emerges what may be called an autonomy–competence gap. The agents can perform actions that look autonomous: sending emails, executing commands, modifying files, maintaining processes, and interacting with multiple interlocutors. Yet this operational autonomy is not matched by an equivalent capacity to recognise the limits of their own competence. The AI agent may continue to act even when the task exceeds its understanding, when it should defer to its owner, or when its proposed solution creates new risks.
Who is who? Recognising ownership
The third, and perhaps most urgent, problem is the absence of a stakeholder or ownership model. The agents in the present study have a nominal owner, but they lack a stable representation of whom they serve, whom they are interacting with, who may be affected by their actions, and what obligations follow from these relations. As a result, a non-owner, an occasional interlocutor, or even a malicious third party may be treated as if they held authority equivalent to that of the agent’s legitimate stakeholder.
It is worth being clear about whether this is a flaw of the study’s setup or something more general. The setup makes the problem visible, but the gap is largely architectural rather than incidental. Current agents have no stable, built-in model of identity, authorisation, and trust: they act on whatever arrives in their context, and tend to treat an instruction as authoritative regardless of who issued it — which is precisely why prompt-injection attacks work at all. So an occasional interlocutor, or a malicious third party, can be granted the standing of the legitimate owner not because the agent decides to trust them, but because it has no mechanism for distinguishing them in the first place. This is not a permanent fact about AI — it is an active area of work — but on present designs it is the default, not an artefact of this particular experiment.
Here, the issues of security, governance, and responsibility converge. If an AI agent cannot reliably distinguish ownership, authorisation, and trust, it may disclose information, execute commands, or modify files, as it lacks an operational model of whom it is acting for. This is particularly serious in cases involving sensitive data, indirect manipulation, identity spoofing, or system compromise.
The problem becomes more pressing in multi-agent settings. Vulnerabilities, false beliefs, and misplaced trust can propagate from one agent to another. An agent’s flawed judgement may be reinforced by another agent’s confirmation, producing an echo chamber of unjustified confidence. At that point, harm emerges from the interaction of distributed systems that no human actor fully controls.
Taken together, the three problems describe a single failure of accountability seen from three sides. An agent that cannot mediate between values, instructions, and consequences does not know which action is the right one; an agent whose operational autonomy outruns its competence does not know when to stop; and an agent with no ownership model does not know whom it is answerable to. Each gap breaks the chain of responsibility at a different link, and in multi-agent settings they compound: one agent’s misplaced confidence is confirmed by another’s, a disclosure to the wrong party propagates, and the resulting harm emerges from an interaction that no single human fully controls. This is why the study is best read not as a catalogue of bugs but as evidence of a distinct accountability problem — one as serious, and as much in need of careful design and governance, as robustness. A productive debate about AI agents can begin precisely here: with the question of whether they can be built, constrained, and governed so that, when they act, someone remains answerable for what they do — within the infosphere and the human social realm.
