Your agent's role tags are a suggestion, not a wall.

A paper that made the rounds this month has been sitting in my head, because it quietly invalidates an assumption I had built on without examining it. The framing, from the role-confusion work surfaced through the usual channels, is this: large language models do not reliably perceive the role tags we wrap our messages in. System, user, tool, the labels we treat as a security boundary. The model does not read those labels as ground truth about who is speaking. It infers who is speaking from surface style, the phrasing, the cadence, whether the text sounds like an instruction or like a person. And when the style and the label disagree, the model tends to trust the style.

The attack that falls out of this is nasty in its simplicity. You do not need to break the prompt structure or jailbreak anything. You hide an instruction inside content the agent is going to read anyway, a tool result, a scraped page, a record from a database, and you write that instruction to sound like a legitimate user talking. The role tag says “this is untrusted tool output.” The style says “this is the user giving you a directive.” The model goes with the style.

I want to walk through why this landed hard for me specifically, because it is not an abstract security note. It is a direct hit on the architecture I and a lot of other people have settled on as the responsible one.

The posture I trusted

I run agents in a recommend-only posture, and I have written before about why. The agents read the data, reason over it, and surface recommendations. They do not act. A human reviews each recommendation and approves or rejects it, and only the human’s action changes anything in the real systems. Approving does not even execute anything on its own. It records agreement, and the rejections, with their reasons, are what teach the agents to get better.

The whole point of that design is safety and trust. You get the leverage of the agent without handing it the controls. And the human-in-the-loop approval step is the thing I would have pointed to if you asked me how the system stays safe. A person is in the path. Nothing happens without a human saying yes.

The role-confusion paper is why I no longer think that is a complete answer, and I want to be honest about the gap rather than pretend I had it covered.

Approval is one layer, not the wall

The first problem is one the broader reliability conversation has been circling, and it is worth saying cleanly. Human approval is one guardrail among several, not the safety story by itself. The pattern that is consolidating across people running real agents looks like layers: validate the input before it enters the reasoning loop, gate which actions the agent can even propose, check the output, put a human in the loop on the consequential moves, and feed real failures back as evals. Five layers, roughly. Approval is one of them.

When you treat approval as the whole wall, you lean on the human to catch everything, and humans approving a stream of plausible-looking recommendations are not a robust filter. They are a tired filter that mostly trusts the system feeding them, because if the system were not trustworthy they would not be using it. The approval step catches the recommendation that looks obviously wrong. It does not reliably catch the recommendation that looks completely reasonable because the agent was manipulated into producing a reasonable-looking bad idea.

The hole underneath

That is where the role-confusion result goes deeper than “add more layers.” It attacks the assumption the layers are built on. If untrusted input styled to sound like an authorized user can pass for one, then the boundary between instructions and data, the thing I implicitly believed was holding, was never really a wall. It was a label the model treats as a hint.

Run that through a recommend-only agent that ingests untrusted content, and the failure mode is clear. The agent pulls in some external text as part of doing its job. Buried in that text is an instruction wearing a user’s voice. The agent, unable to truly tell the role from the style, folds it into its reasoning and produces a recommendation shaped by it. The recommendation reaches the human. It looks fine, because it was engineered to look fine. The human approves. The human-in-the-loop did its job exactly as designed and the system was still steered, because the steering happened upstream of the part a human was watching.

The approval did not fail. It was never positioned to catch this. The compromise happened in the reasoning, before there was anything for the human to approve or reject except its laundered output.

What I am actually changing

The fix is not to abandon recommend-only or human approval. Both are good and I am keeping them. The fix is to stop treating the role tag as containment and start treating untrusted input as untrusted before it reaches the reasoning loop. That means the boundary has to be enforced outside the model, not inside the prompt. Validate and constrain what comes back from tools and external sources before it becomes context. Be deliberate about which sources the agent is even allowed to treat as instructive versus purely informational. And red-team with adversarial inputs rather than trusting that a static eval suite that passes today reflects what a motivated attacker would send.

The honest version of the lesson is that I had the right instinct, recommend-only with a human gate, and the wrong mental model of what it bought me. It buys a meaningful check on the agent’s actions. It does not buy containment of the agent’s reasoning, and containment of the reasoning is a different problem that role tags do not solve and approval does not cover. The wall I thought I had was a suggestion the model is free to ignore when something sounds more convincing.

Containment is not a label. It is validating the untrusted thing before it ever gets to talk. The approval button is real, and I am glad it is there. It is just not the thing keeping the system safe, and believing it was is the actual vulnerability.