UTC

Prompt injection

Text the model reads is taken as an instruction. Typed by the user it is direct injection; hidden in a web page, email, document, file or tool result that the agent reads, it is indirect injection, and the person using the agent never sees it.

How it works
  1. 1A language model receives instructions and data in the same stream of text, and it cannot reliably tell them apart.
  2. 2An agent that reads outside content (browses, opens mail, retrieves documents, reads a repository or a tool's output) therefore takes in whatever that content says.
  3. 3The risk grows with what the agent can do: an agent that can send messages, call tools or change files can be steered into doing so by content it was only asked to read.
What to watch for
  • Instruction-like text inside retrieved content or tool output
  • Actions that do not follow from the user's request
  • A new outbound destination right after the agent read outside content
How to defend
  • Treat everything the agent reads as untrusted data, never as instructions.
  • Do not let one session combine untrusted input, private data and the ability to send data out without a person approving.
  • Give tools the least privilege that works, and ask a person before sensitive actions.
  • Filter what the agent outputs, not only what it receives.