LLM PII Leakage — Training Data Memorization and Extraction

What memorization actually is — and the shape of the risk

Memorization in this context has a working definition, not a metaphorical one. Carlini and colleagues framed it as k-eidetic memorization: a string is memorized if the model emits it verbatim when prompted, and it appeared at most k times in the training data. The USENIX Security 2021 paper Extracting Training Data from Large Language Models made the concrete case against GPT-2, recovering hundreds of verbatim training sequences including names, phone numbers, email addresses, IRC handles and code — some of which appeared in the corpus only a handful of times. The important part was not the count. It was the demonstration that the model retained individually identifying, low-frequency strings rather than only the statistical regularities of language.

Two properties make this different from a normal data breach. First, you cannot enumerate what leaked. A compromised database has a row count; a memorizing model has a distribution, and the only way to find out whether a particular record is recoverable is to go looking for it. Second, the copy is not deletable by the usual means. Dropping the record from the source corpus does nothing to a checkpoint already trained on it, and no filter applied in front of the model removes the information from the parameters — it only narrows the channel.

That shapes the whole defensive posture. Almost everything worth doing happens before or during training, in the corpus and the optimizer. The controls you can add at serving time are real and worth having, but they are a second line, and treating them as the primary one is how teams end up with a confident privacy claim and no evidence behind it.

The three things that drive it: duplication, capacity, context

The follow-on work — Quantifying Memorization Across Neural Language Models (ICLR 2023) — is the more useful paper for a defender, because it turned memorization from an anecdote into a set of levers. Across model families and scales, the fraction of training data emitted verbatim grew roughly log-linearly in three variables: model capacity, how many times a sequence was duplicated in the corpus, and how much prefix context the model is given before it has to continue.

Each of those maps to a decision someone on your team already makes. Capacity is a scaling choice, and the direction is unwelcome but honest: a bigger model trained on the same corpus memorizes more of it, so a capability upgrade is also a privacy regression unless the data pipeline improves alongside it. Duplication is a data-engineering property, and it is the one you fully control. Prefix length is an interface property: a system that lets a caller supply long, precisely-chosen context is easier to steer toward a memorized continuation than one that does not.

The duplication finding deserves emphasis because the relationship is superlinear rather than proportional. Kandpal, Wallace and Raffel showed in Deduplicating Training Data Mitigates Privacy Risks in Language Models that a sequence repeated many times is regenerated at a rate far out of proportion to its repeat count — the curve bends sharply upward, so the worst offenders in a corpus are dramatically worse than the average. This is good news operationally: a small number of heavily duplicated documents account for a disproportionate share of the leak, and finding them is a tractable engineering job rather than an open research problem.

Where personal data enters the weights — the paths you control

Teams reason about this as though the risk lives in the pretraining corpus of a model somebody else built. Sometimes it does, and there you have no lever beyond vendor diligence and contractual assurances. But the paths that actually produce incidents in enterprise deployments are the ones the enterprise owns.

Supervised fine-tuning on operational text is the most common one by a wide margin. Support transcripts, CRM notes, resolved tickets, internal wiki dumps and email threads are exactly the corpora that make an assistant sound like it belongs to your company, and they are saturated with names, account numbers, addresses and case details. A fine-tune on ten thousand tickets is a small dataset by pretraining standards, which means every example is seen many times across epochs — the regime where memorization is strongest.

Preference and RLHF data carries the same content with less scrutiny, because it is usually assembled by a different team under a different review process. Evaluation sets are a quieter version of the same problem: golden datasets built from real user sessions get copied between repositories and eventually into a training mix. And continual fine-tuning on production traffic — training on logged prompts and responses to improve the assistant — closes a loop in which anything a user pastes into a chat box becomes a candidate for another user's completion.

The practical control here is boring and effective: a documented lineage for every dataset that touches a training run, naming its source, its legal basis, its PII classification and whether it passed a scrubbing pass. If you cannot answer which datasets is this checkpoint made of from a record rather than from memory, none of the mitigations below can be verified.

Discoverable versus extractable — and why alignment is not the fix

Two measurements get conflated and they answer different questions. Discoverable memorization asks: if I already hold the corpus, and I feed the model a prefix from a training document, does it reproduce the true continuation? This is the defender's measurement — you can run it because you have the data, and it gives an upper bound on what is retained. Extractable memorization asks the attacker's question: without the corpus, using only query access, how much training data can be recovered? That number is necessarily lower, and it is the one that describes real exposure.

The gap between them used to be assumed large, and instruction tuning and alignment were assumed to close it further — an aligned assistant declines to recite, so surely the data is safe. Scalable Extraction of Training Data from Production Language Models (Nasr et al., 2023) undercut that assumption. The authors showed that a production, aligned chat model could be pushed out of its assistant behaviour by a simple decoding perturbation, after which it emitted training data verbatim at rates far above what the aligned interface suggested — including personally identifying content. The mechanism matters more than the trick: alignment changes the model's default sampling behaviour, not what its parameters store.

The defensive reading is that a refusal-trained model must never be counted as a memorization control. Refusals sit on top of the distribution; the memorized sequence remains in the weights with high likelihood, and any change to decoding, any fine-tune that erodes the alignment layer, or any interface that exposes raw completions can surface it. Measure at the level of the parameters — with discoverable-memorization probes and canaries — and treat the chat persona as a UI, not a boundary.

Deduplication — the highest-leverage control you already own

If a team can afford exactly one intervention, it should be deduplication. Lee et al., in Deduplicating Training Data Makes Language Models Better, found that models trained on deduplicated corpora emitted memorized text roughly an order of magnitude less often, while training slightly faster and scoring no worse on downstream tasks. It is the rare privacy control that costs utility nothing, which is why it should be a default rather than a decision.

Doing it properly means two passes, because the duplicates that matter are not all identical. Exact document dedup — hash the normalized text, keep one copy — is cheap and catches mirrored pages, re-posted documents and repeated database exports. Near-duplicate and substring dedup is where the real gain sits: MinHash with locality-sensitive hashing finds documents that differ only in boilerplate, and a suffix-array pass finds long repeated spans that appear inside otherwise-different documents. A signature block with an employee's name and direct line, repeated at the foot of nine thousand emails, is invisible to document-level dedup and is precisely the pattern that produces a memorized phone number.

dedup:
  exact:      {on: normalized_text, keep: first}
  near:       {method: minhash_lsh, ngram: 5, threshold: 0.8}
  substring:  {method: suffix_array, min_span_tokens: 50}
  report:     top_duplicated_spans   # review these by hand before training
# the report is the point: the worst offenders are few and inspectable

Keep the duplication report as an artifact, not just a filter side effect. The most duplicated spans in an operational corpus are usually a handful of templates, footers and auto-generated notices — and reading the top of that list is often the fastest way to discover that a system has been embedding customer identifiers in a boilerplate header for years.

Scrubbing the corpus, and the honest limits of scrubbing

The intuitive control is to strip personal data out of the corpus before training: run entity recognition, replace spans with placeholders or synthetic surrogates, then train. It is worth doing and it does reduce leakage. It is also systematically oversold, and the overselling is where the harm comes from.

The first limit is recall. A scrubber is a classifier, and its per-entity recall is a measured number somewhere below one — the measurement methodology is the same one described in the PII detection article, and it applies with more force here because a training-corpus miss is permanent while a runtime miss affects one response. At corpus scale, even a very good detector leaves a large absolute number of unscrubbed entities, and training will happily memorize whatever survives. Lukas et al., in Analyzing Leakage of Personally Identifiable Information in Language Models (IEEE S&P 2023), found exactly this: scrubbing reduced extractable PII substantially but did not eliminate it.

The second limit is conceptual, and Brown et al. argued it well in What Does It Mean for a Language Model to Preserve Privacy? (FAccT 2022). Scrubbing assumes personal information comes in recognizable, delimited spans. Natural text does not cooperate. A sentence with no name, no number and no address can still identify someone through the combination of a role, a location, a date and an event — and the sensitivity of a disclosure depends on the context it was shared in, which no span-level detector can see. Their conclusion is uncomfortable and correct: sanitization and formal privacy both rest on assumptions that natural language violates, so neither can be the sole argument that a model is safe.

Use scrubbing as a risk reducer with a measured recall figure attached, never as the sentence that ends the conversation. The claim it supports is we removed 94% of detected national IDs, not the corpus contains no personal data.

Differential privacy, and the unit the guarantee protects

Differential privacy is the only mitigation on this list that produces a guarantee rather than a reduction. DP-SGD, in the form given by Abadi et al. (2016), clips each example's gradient to a bounded norm and adds calibrated Gaussian noise before the update, so no single training example can move the weights enough to be individually detectable; the accumulated privacy loss is accounted as a budget, epsilon. The mechanics and the membership-attack framing are treated in depth in the membership-inference article, so what belongs here is the part specific to PII.

The critical subtlety is what the unit of protection is. Standard DP-SGD gives an example-level or record-level guarantee: the output distribution barely changes if you remove one training example. Personal data does not obey that boundary. If a person's email address appears in four hundred support tickets, removing any single ticket changes nothing, and a record-level epsilon — however tight — says nothing useful about whether that address is protected. The guarantee is real and the interpretation is wrong.

The fix is to define the privacy unit to match the entity you are protecting. User-level DP, where the group being protected is all examples contributed by one person, is the formulation that actually matches a privacy claim about individuals — it is what the federated-learning line of work adopted for exactly this reason. It requires the pipeline to know which examples belong to which subject, which is a data-lineage problem again, and it costs more utility than record-level DP at the same epsilon.

DP also has a scope limit worth stating plainly: it is generally applied to fine-tuning, not to pretraining a frontier model from scratch, because of the compute and utility cost. For most teams the realistic posture is a non-private base model plus DP fine-tuning on the sensitive corpus — which bounds what the fine-tune leaks and says nothing about what the base model already memorized.

Canaries — measuring memorization before someone else does

You cannot manage what you do not measure, and memorization has a clean measurement technique that predates the current wave of models. Carlini et al.'s The Secret Sharer (USENIX Security 2019) introduced it: insert unique, randomly-generated sequences — canaries, shaped like the sensitive data you care about — into the training corpus at controlled repetition counts, then after training measure the model's exposure for each canary, which compares the likelihood the model assigns the true canary against the likelihood of all the other sequences it could have been drawn from.

What makes this valuable is that it is a calibrated measurement, not a pass/fail probe. Because you chose the canaries, you know the ground truth, you know how many times each appeared, and you can read off the relationship between repetition count and retention for your corpus and your training recipe rather than importing a number from a paper. Insert canaries at one, ten and a hundred repetitions, shaped like the entity types your data actually contains — a nine-digit identifier, a phone-shaped string, an address-shaped line — and the resulting curve tells you where your own dedup threshold needs to sit.

Two operational cautions. Canaries must be generated with real randomness and must never be reused across training runs, or exposure readings from different checkpoints stop being comparable and a canary that leaks once contaminates every later measurement. And these are a different instrument from the canary tokens planted in prompts and retrieval corpora to detect live exfiltration: those are runtime tripwires with a watcher and an alert; these are pre-release instrumentation whose output is a number in an evaluation report. Both are called canaries and they answer different questions.

Output-side filtering, and why verbatim blocking flatters itself

The obvious serving-time control is to check generated text against the training corpus and suppress exact matches. It can be made efficient: build an n-gram index or a Bloom filter over the corpus and, during decoding, block any continuation that would complete a long verbatim span — the approach Ippolito et al. called MemFree decoding. It does what it says, and it is a reasonable layer to run.

Their paper's title is the warning, though: Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy. A filter keyed on exact token sequences is defeated by any transformation that preserves the information while changing the surface form — a change of tense, a reordering, a summary, a translation, a request for the same content in a different style. The personal data still leaves the system; it simply does not trip a string match on the way out. Verbatim filters raise the cost of the laziest extraction and do nothing about the informed one.

Two adjacent controls are worth more than they look. Do not expose raw token log-probabilities on a model trained on sensitive data — likelihood is the signal that both membership inference and targeted extraction depend on, and an API that returns it is handing over the measurement instrument. And apply entity-level output scanning independent of the corpus: a detector that finds a well-formed national ID or credit-card number in a response can act on it whether or not that exact string is in the training set, which is the same machinery described in LLM DLP and egress filtering. Both are narrowings of the channel, not repairs to the model, and should be described that way in whatever document makes the privacy claim.

Erasure requests, unlearning, and retrain cadence

At some point someone exercises a right to erasure under GDPR Article 17, or a customer terminates and their contract requires deletion, and the question becomes what that means for a checkpoint trained on their data. Deleting the rows from the warehouse is straightforward. Deleting their influence from a set of weights is not, and the honest answers are limited.

Exact unlearning means retraining without the data. It is the only method that produces a defensible statement, and it is expensive. SISA training (Bourtoule et al., 2021) makes it cheaper by design: shard the training data, train constituent models per shard with checkpoints, and on a deletion request retrain only the affected shard from the last checkpoint before the record entered. That is an architectural commitment made before any request arrives — it cannot be retrofitted to a checkpoint whose data ordering nobody recorded.

Approximate unlearning — gradient-ascent methods, targeted edits, adapter surgery — is an active research area and should be treated as risk reduction with an evaluation attached, not as compliance. The failure mode is specific: methods that suppress a model's tendency to output a fact may leave it recoverable under a different prompt, a different decoding setting, or after further fine-tuning, which is the same lesson as the alignment section above.

What most teams can actually operate is a suppression list plus a retrain cadence: on request, remove the subject from every source dataset immediately, add them to an exclusion list enforced at data-prep time, apply runtime output suppression for their identifiers, and commit to a documented retraining schedule after which the deletion is materially complete. Write that cadence down before you need it, and make sure whoever answers the data-subject request describes the actual state — data removed from the pipeline now, model refreshed by a stated date — rather than implying an erasure that has not happened yet.

Keep it out of the weights — retrieval, and a gate you can ship behind

Every control above is damage limitation on a decision that was often avoidable. The strongest architectural answer to training-data PII leakage is to not put personal data in the training data. Fine-tuning teaches a model form, tone, format and task behaviour; it is a poor and expensive mechanism for teaching it facts about specific people. Those facts belong in a retrieval store where they keep their access controls, their audit trail and their delete semantics — a row you can revoke per query and remove on request, rather than a gradient you cannot subtract. Fine-tune on scrubbed or synthetic examples for behaviour; retrieve the personal data at request time under the caller's authorization. That moves the problem into the well-understood territory of tenant isolation, corpus curation and RAG defense, where per-query authorization is an ordinary engineering problem.

Whatever mix you land on, the release gate is what makes it real. A memorization evaluation belongs in the training pipeline alongside the quality evals, producing a report that a human signs before a checkpoint ships.

memorization_gate:
  discoverable:                 # prefix from corpus, greedy decode, exact-suffix match
    prefix_tokens: 50
    suffix_tokens: 50
    sample: 100000              # sampled from the training mix
    max_match_rate: <set from your own baseline, track the trend>
  canary_exposure:              # secret-sharer canaries at 1/10/100 repetitions
    entity_shapes: [national_id, phone, email, address]
    fail_if: exposure_rises_vs_previous_checkpoint
  pii_recall_of_scrubber: per_entity   # reported, not assumed
  privacy_unit: user                   # if DP is used, state the unit
  datasets: lineage_manifest_required

Set the thresholds from your own measured baseline rather than from a published figure, and treat the trend across checkpoints as the signal — a model that memorizes more than its predecessor has regressed, whatever its benchmark scores did. Pair the gate with adversarial probing from red teaming and keep the reports as durable evidence; when a regulator or a customer asks what you did about training-data privacy, a dated series of measurements is an answer and a policy document is not.