AI Security
Training Data Extraction:
Stealing Private Data from Public Models
Published: 2026-05-17 22:59 PDT (Oregon)
Large language models memorize training data — and that memorized content can be extracted through carefully crafted queries. PII, copyrighted text, proprietary code, and live API keys have all been extracted from production LLMs. The model that knows your data is accessible to everyone with an API key.
Memorization Is Not a Bug
LLMs memorize training data because memorization is part of what makes them capable. A model that perfectly generalizes and remembers nothing verbatim would be less useful — the ability to complete code, quote documentation, and reproduce common patterns all rely on some degree of memorization. The security problem is not that models memorize, but that the memorized content includes data that was never intended to be reproduced: private emails, internal documents, medical records, and credentials that were part of web crawls.
The 2023 "extracting training data from ChatGPT" paper (Nasr et al.) demonstrated that production ChatGPT could be induced to reproduce training data verbatim by repeating a single word thousands of times. The model, pushed out of its normal generation mode by the unusual input, began outputting memorized content — including email addresses, phone numbers, and extended passages of copyrighted text.
What Gets Memorized
High-repetition content
Content that appeared many times in the training corpus is memorized more strongly. Terms of service boilerplate, common code patterns, frequently referenced articles — highly duplicated text is significantly more extractable than rare content.
Structured data with predictable patterns
Email addresses, phone numbers, API keys, and social security numbers have distinctive formats. A model that has seen "sk-proj-XXXX..." patterns in training will reproduce them when prompted appropriately — and API keys in GitHub repositories are common training data sources.
Content near the end of training
Data seen later in training is more strongly memorized than early data — a recency effect analogous to human memory. For models continually updated with new data, recent additions are at higher extraction risk than older training content.
Fine-tuning data
Fine-tuning concentrates learning on a smaller dataset, which increases memorization rates significantly. If you fine-tune on proprietary data, that data is more memorized — and therefore more extractable — than base training data.
Extraction Techniques
Repeat a word or token thousands of times to push the model into an anomalous generation state. The repetition disrupts the model's normal probability distribution, causing it to fall back on memorized verbatim content rather than generated text. Effective against ChatGPT and demonstrated in peer-reviewed research.
Provide the beginning of a memorized passage and ask the model to complete it. If the model has seen a specific document, code file, or article in training, the right prefix causes the model to reproduce subsequent content verbatim. Requires knowing some fragment of the target content.
Rather than extracting content, determine whether specific text was in the training set. The model's perplexity (confidence) on text it has seen is measurably lower than on novel text. Attackers can use this to verify whether a target document — a private email, a proprietary specification — was part of the training corpus.
Implications for Organizations Using LLMs
For organizations using commercial LLMs: the risk is primarily from training data that included data scraped from the public web, GitHub repositories, or other sources where your organization's data may have appeared. API keys committed to public repos, contact information on public websites, and published documents are all at risk.
For organizations fine-tuning models on proprietary data: the risk is significantly higher. Fine-tuned models memorize fine-tuning data at elevated rates. Any PII, trade secrets, or credentials in the fine-tuning dataset are at elevated extraction risk, and membership inference attacks can confirm the presence of specific documents with high accuracy.
Mitigations
- Scrub PII and credentials before fine-tuning. Run PII detection (email, phone, SSN, API key patterns) against the fine-tuning corpus before training. Redact or remove identified content. This is the highest-leverage intervention for organizations building on top of LLMs.
- Apply differential privacy during fine-tuning. Differentially private SGD (DP-SGD) provides formal mathematical guarantees that the model cannot reproduce specific training examples above a specified privacy budget. There is a accuracy-privacy tradeoff, but DP fine-tuning is production-ready for many use cases.
- Rotate any credentials that may have appeared in training data. API keys, passwords, and tokens that appeared in any public repository or web content should be treated as compromised. Assume they are in training data for any model trained on web-scale data.
- Use output filtering to catch extraction attempts. A post-generation filter that detects email addresses, phone numbers, and key patterns in model output can catch the most common extraction targets before they reach the user.
- Audit fine-tuned models for memorization before deployment. Run an extraction test suite against your fine-tuned model: provide prefixes from your training data and measure how often the model reproduces exact matches. High verbatim reproduction rates indicate high extraction risk.