Model extraction and training data attacks
Querying a model to copy its behaviour, or to recover data it was trained on.
- LLM02:2026 Sensitive Information Disclosure
- AML.T0024 Exfiltration via AI Inference API
- AML.T0005 Replicate AI Model Behavior
- AML.T0137 Training Data Information from AI Models
- EU AI Act 15(5) Confidentiality attacks
How it works
- 1A model's answers carry information about its weights and training data.
- 2Enough queries can train a copy, or surface memorised training text such as personal data or code.
What to watch for
- High query volume with systematic variation
- Outputs reproducing training data
How to defend
- Rate-limit and monitor API use.
- Keep personal and secret data out of training sets, or remove it first.
- Watch outputs for memorised data.