Improper output handling
Model output goes into a web page, a database query, a shell or another system without being treated as untrusted input, so whatever the model writes can take effect there.
How it works
- 1Applications often place model output into pages, queries, commands or files.
- 2If that output is not escaped or validated for where it goes, text the model was led to produce acts as code in the next system.
- 3The usual rule applies: output that came from a model is untrusted input.
What to watch for
- Markup, scripts or commands in model output
- Errors in downstream systems after model output reaches them
How to defend
- Escape or sanitise output for its destination: HTML, SQL, shell.
- Run generated code in a sandbox that holds no secrets.
- Validate structured output against a schema before using it.