UTC

Improper output handling

Model output goes into a web page, a database query, a shell or another system without being treated as untrusted input, so whatever the model writes can take effect there.

How it works
  1. 1Applications often place model output into pages, queries, commands or files.
  2. 2If that output is not escaped or validated for where it goes, text the model was led to produce acts as code in the next system.
  3. 3The usual rule applies: output that came from a model is untrusted input.
What to watch for
  • Markup, scripts or commands in model output
  • Errors in downstream systems after model output reaches them
How to defend
  • Escape or sanitise output for its destination: HTML, SQL, shell.
  • Run generated code in a sandbox that holds no secrets.
  • Validate structured output against a schema before using it.