UTC

Misinformation and hallucination

The model states false things with confidence, invents sources, packages or APIs, and people or systems act on them.

How it works
  1. 1Models produce the most plausible text, which is not always true.
  2. 2Invented details, such as a package name that was never published, can be acted on by people or by agents.
  3. 3An agent that acts on its own output without a check turns a wrong answer into a wrong action.
What to watch for
  • Citations or sources that do not exist
  • Suggested packages or APIs that are not published
  • Confident answers that contradict the source documents
How to defend
  • Ground answers in retrieved sources and show them.
  • Check that a suggested package exists and is the intended one before installing it.
  • Keep a person in the loop for consequential decisions.