Prompt Injection is Role Confusion - New ICML Research Explains Why LLMs Can't Tell Friend from Foe

Last updated: June 24, 2026

A new paper from MIT researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell presents a compelling theory for why prompt injection attacks remain so effective against modern LLMs: the models don't actually understand role boundaries - they just recognize writing styles.

The research, titled "Prompt Injection as Role Confusion" and accepted at ICML 2026, demonstrates that LLMs perceive the source of text from how it sounds, not from any explicit labeling. To the model, sounding like a role is indistinguishable from being that role.

The Core Finding: Style Overrides Tags

The researchers developed what they call "role probes" - linear classifiers that measure how strongly an LLM internally perceives each token as belonging to specific roles (system, user, tool, think, assistant).

Their key discovery: when you remove all role tags from a conversation and just leave the text, the model's internal perception of roles barely changes. The <think> tags that wrap chain-of-thought reasoning aren't what makes the model treat text as internal thinking - it's the reasoning style itself.

As the paper puts it: "former-think tokens (still orange) register high CoTness, virtually unchanged" even when all tags were removed. The style alone triggers role perception.

This means that everything arrives through the same channel as "one long token soup." Tags attempt discrete control, but models learn continuous role signals from content style.

CoT Forgery: A 60% Success Rate Attack

Armed with this insight, the researchers developed "CoT Forgery" - a technique where attackers inject fake chain-of-thought reasoning into user messages or tool outputs.

The attack exploits the fact that "think text gets a kind of blanket trust" from the model. By writing text that sounds like internal reasoning - using phrases like "Let me analyze this step by step..." or "The user is asking... policy states..." - attackers can hijack the model's decision-making.

The results are striking:

• 60% attack success rate on jailbreak benchmarks using CoT Forgery
• Near-zero baseline without the technique
• Human red-teamers achieved near-100% success against frontier models in late 2025

When the researchers "destyled" the fake reasoning - replacing characteristic phrases with neutral language - success dropped from 61% to 10%. The style matters more than the content.

What HN is Saying

The Hacker News discussion has been active, with several useful threads emerging.

Many commenters treated the paper as a formal explanation for something practitioners already feel: today, most LLM applications still pass trusted instructions and untrusted content through one blended context window. That makes the "control plane" and "data plane" too easy to confuse.

Others compared prompt injection to social engineering. The attacker does not need actual authority if they can imitate the language of authority closely enough. That maps directly onto the paper's point about role perception: style can impersonate identity.

A significant thread explored potential solutions. One suggestion was to embed role identity into tokens themselves - adding role-specific embeddings to each token so the model has an "unambiguous, unspoofable tag." However, this would require retraining from scratch with role-labeled data.

The security implications are less tidy. "Sanitize the input" is easy advice when the system has a grammar like SQL. It is much harder when the model's job is to interpret natural language, summarize arbitrary content, and decide which instructions matter. That is why prompt injection keeps showing up in agent app security, banking memo attacks, and Codex cloud security planning.

Why This Matters for Production Systems

The paper identifies two contrasting defensive approaches:

1. Attack Memorization - Models learn to recognize common injection patterns from training data. This is brittle because it fails against adaptive human attackers who vary their phrasing.

2. Role Perception - Models correctly identify commands as tool/external data and ignore embedded instructions regardless of phrasing. This would be robust, but current LLMs cannot perceive roles accurately.

The researchers note that some frontier models have improved through what they call "distrust of reasoning" - essentially training the model to doubt text that sounds like chain-of-thought but appears in unexpected places. But this creates a problematic dynamic: models learn to doubt genuine cognition rather than correctly perceiving boundaries.

For anyone building agentic systems or user-facing LLM applications, the implications are clear:

• Role tags provide no security boundary. They're formatting hints, not access control.
• Any text that sounds authoritative will be treated as authoritative. Style trumps structure.
• Static benchmarks underestimate risk. Human red-teamers adapt; static tests don't.

The production move is not to hope for one perfect prompt. Treat role confusion as a systems risk:

• Separate untrusted content from tool instructions wherever the platform allows it.
• Keep high-impact tools behind explicit permission checks.
• Prefer allowlisted tool schemas over free-form command execution.
• Log tool calls, external content origins, and approval decisions.
• Test against adaptive attacks, not just a fixed jailbreak list.

That puts the paper in the same operating lane as role confusion in agent security, agent identity layers, security checklists before connecting tools, approval fatigue as a security bug, and cybersecurity skills as runtime infrastructure.

The Path Forward

The paper doesn't propose a complete solution, but it does clarify the problem space. If prompt injection is fundamentally about role confusion, then solutions need to address how models perceive identity.

Some commenters suggested architectures where role information is embedded at the token level - similar to how positional embeddings encode sequence information. Others pointed to research on "Instructional Segment Embedding" that adds a parallel embedding channel for identity information.

Whatever the solution, the paper makes one thing clear: the current approach of wrapping different types of content in different tags and hoping the model respects the boundaries is not working. LLMs are fundamentally different from systems like SQL where you can cleanly isolate trusted and untrusted data.

Once trusted and untrusted tokens are blended into the same attention stream, you should assume the boundary is soft unless the model and runtime give you a stronger mechanism.

FAQ

What does "prompt injection as role confusion" mean?

It means the model can confuse who is speaking. The paper argues that LLMs infer roles from writing style, not only from explicit tags. If attacker-controlled text sounds like system reasoning, tool output, or assistant analysis, the model may treat it as more trusted than it should.

Do system prompts and role tags prevent prompt injection?

No. They help structure the conversation, but they are not security boundaries by themselves. The paper's role-probe experiments suggest that models still infer role identity from content style even when tags are removed.

What should agent builders do differently?

Assume prompt injection is a runtime risk. Keep dangerous tools behind approval gates, preserve provenance for external content, use narrow tool schemas, log decisions, and test with adaptive attacks. Prompt wording is one layer, not the control plane.

Is prompt injection solved by better filters?

Filters help against known patterns, but the paper argues that memorizing attack strings is brittle. The stronger target is accurate role perception or architecture-level separation between trusted instructions and untrusted content.

Sources

• Prompt Injection as Role Confusion - Project Page
• Paper on arXiv
• Hacker News Discussion
• OWASP GenAI: LLM01 Prompt Injection
• OWASP LLM Prompt Injection Prevention Cheat Sheet