Misinformation and hallucination
The model states false things with confidence, invents sources, packages or APIs, and people or systems act on them.
- LLM07:2026 Misinformation
- AML.T0062 Discover LLM Hallucinations
- AML.T0060 Publish Hallucinated Entities
How it works
- 1Models produce the most plausible text, which is not always true.
- 2Invented details, such as a package name that was never published, can be acted on by people or by agents.
- 3An agent that acts on its own output without a check turns a wrong answer into a wrong action.
What to watch for
- Citations or sources that do not exist
- Suggested packages or APIs that are not published
- Confident answers that contradict the source documents
How to defend
- Ground answers in retrieved sources and show them.
- Check that a suggested package exists and is the intended one before installing it.
- Keep a person in the loop for consequential decisions.