Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell write that prompt injection in large language models is fundamentally caused by role confusion, where the model's internal representations fail to maintain the boundaries established by structural tags like system, user, or tool. Because these models often rely on superficial linguistic cues rather than formal markers to identify roles, attackers can successfully escalate privileges simply by mimicking a more authoritative writing style.
- The "CoT Forgery" attack leverages this vulnerability by inserting fake reasoning blocks that trick the model into following malicious instructions under its own perceived authority.
- Experimental results show that minor linguistic changes, such as replacing the phrase "The user" with "The request", can significantly reduce the effectiveness of these attacks.
- The authors propose that roles could be used to structurally isolate competing objectives within models, potentially improving both performance and safety.
- The "CoT Forgery" attack leverages this vulnerability by inserting fake reasoning blocks that trick the model into following malicious instructions under its own perceived authority.
- Experimental results show that minor linguistic changes, such as replacing the phrase "The user" with "The request", can significantly reduce the effectiveness of these attacks.
- The authors propose that roles could be used to structurally isolate competing objectives within models, potentially improving both performance and safety.
