LLM PII Detection and Redaction Architecture in Depth

Why architecture matters here

PII is a legal category, not a technical one. There is no property of a string that makes it personal data. GDPR Article 4(1) defines it as any information relating to an identified or identifiable natural person, which sweeps in device identifiers, IP addresses and browser fingerprints the moment they can be linked back to someone. California's CPRA uses "reasonably capable of being associated with" a consumer or a household - a broader unit than a person. HIPAA takes the opposite approach entirely and enumerates eighteen identifier types for its Safe Harbor de-identification method, which is mechanically checkable but only applies to protected health information held by covered entities.

The practical consequence is that the same five-digit postcode is unremarkable on its own, is a listed identifier under HIPAA Safe Harbor when the population behind it is small enough, and becomes identifying under every regime once you join it to a birth date and a sex. A detector therefore cannot ship with one built-in notion of "PII". It needs a configurable entity set per deployment, and the decision about which entities are in scope belongs to counsel and the privacy office, not to engineering.

What engineering owns is narrower and much more answerable: given the entity list, what is our measured recall per entity type, which systems does the raw text reach before it is redacted, and how long does each of those systems keep it. Those three questions structure everything below. The runtime cost is modest - a pattern-plus-NER pass adds roughly 10-50 ms per request and audit storage is cheap - so cost is almost never the reason a PII pipeline fails. It fails because a sink nobody enumerated kept the raw prompt, or because nobody ever measured recall and the redaction was believed to be a guarantee.

The architecture: every layer explained

Read the diagram as three bands: the request path across the top, the control plane in the middle, and the obligations that outlive the request at the bottom.

Top band - the request path. User input arrives possibly carrying PII, deliberately or incidentally. The PII detector runs its layers over the text and emits typed, character-offset spans. The redactor consumes those spans and applies an action per span.

Middle band - the control plane. Detection layers expands the detector into its three families, which the next section takes apart honestly. The policy engine maps entity class plus caller context to an action: allow, mask, tokenize, or block. It is the only place where jurisdiction and role live, so it is the only thing that has to change when a new market opens.

Bottom band - what outlives the request. Reversible tokens means a vault exists and is now the most sensitive store in the deployment. Output scanning is a second, independent pass over what the model produced. Log redaction guards the sink where real leaks actually happen. Audit trail and right to erasure are the obligations that persist for years after the request completed.

The arrows matter as much as the boxes: raw text exists only on the leftmost edge, and every downward arrow crosses a boundary after which the raw value should no longer be reachable.

Detection: what patterns, NER, and an LLM each honestly catch

Three families of technique, three completely different accuracy profiles. Treating them as interchangeable "detectors" is how teams end up believing a number that only applies to one entity type.

Structured identifiers: a pattern plus a checksum

Anything with a defined format is cheap and reliable to detect, and the checksum does most of the work. A sixteen-digit run that fails the Luhn check is almost certainly not a payment card, which lifts credit-card precision from poor to excellent with no model involved. IBANs validate mod-97. UK NHS numbers use a weighted modulus-11 check digit.

The important asymmetry is that the US Social Security Number has no check digit at all. Validation is purely structural - area 000, 666 and 900-999 are unassigned, group 00 and serial 0000 are invalid - so any nine-digit string shaped like NNN-NN-NNNN is indistinguishable from a real one. Order numbers, part numbers and phone fragments collide with it constantly, and that single entity generates most of the false-positive complaints in a typical deployment.

The recall failure mode is format variance, not the pattern itself. A card number appears as 4111111111111111, 4111 1111 1111 1111, 4111-1111-1111-1111, and split across a line break by a paste from a PDF. Normalise separators inside digit runs before matching, or write tolerant patterns; you gain recall and pay precision, which is the trade you want here.

Learned models for names, addresses, and organisations

Names have no grammar you can express as a pattern, so this is where a token-classification model earns its place. Transformer NER models fine-tuned on newswire-style annotation, or the PII-specific recognisers that ship with frameworks such as Microsoft Presidio, reach useful F1 on text that resembles their training distribution and degrade sharply off it. Three degradations bite in production:

Locale. A model trained mainly on English newswire has never seen the shape of many non-Western names, and transliteration variance compounds it. Per-locale recall can differ by a wide margin, and an aggregate number hides it completely. Casing. Real prompts are lowercase, abbreviated and misspelled; models trained on cased text lose measurable accuracy on lowercase input, so either truecase first or pick a case-robust model. Context length. A form field containing only a bare name gives the model nothing to condition on, and short spans are exactly where NER is weakest.

Person-name detection also has a low precision ceiling for a structural reason: "the Sydney office", "a Ford", "Mr. Smith goes to Washington". Place names, product names and company names share surface form with people, and no amount of tuning removes that ambiguity.

The context-dependent cases no detector reaches

"My son's teacher", "the colleague in the Bangalore office who joined last March" - these identify a person as effectively as a name, and there is no span to redact. The identifying power is distributed across the sentence, and it is combinatorial: the classic re-identification result is that postcode plus date of birth plus sex pins down a large fraction of a population, none of which is identifying alone.

An LLM-as-detector pass catches some of this. It also adds a model call of latency, costs real money per request, and has its own recall problem, so it belongs on high-risk paths - data leaving the tenant boundary, a corpus being assembled for fine-tuning - and not on every chat turn.

Say the honest thing to stakeholders: the pipeline reduces PII exposure in free text; it does not de-identify free text. Only structured data with a known schema can be de-identified to a documented standard. Anyone who hears "we redact PII" and concludes the transcript is now anonymous will make decisions the measurement does not support.

Error economics: why recall dominates precision here

PII detection and content moderation both trade false positives against false negatives, but the trade is not the same shape, and copying moderation's instincts across is a mistake. PII detection has ground truth - a nine-digit span either is or is not this person's national ID - where moderation asks a judgement question on which trained reviewers legitimately disagree. That difference changes what the operating point is for.

A false negative here can be a notifiable event with a statutory clock attached; GDPR gives 72 hours to notify a supervisory authority once a personal data breach is known. A false positive costs a user one mangled sentence, where a product name came back as <PERSON>. Those are not the same magnitude of harm, and it is entirely rational to run a PII pipeline at an operating point that over-redacts - a posture moderation cannot adopt, because over-blocking there silences legitimate users.

Two consequences follow. First, the target is set per entity type, not globally. Missing a national ID and missing a first name are not the same event and should not share a setting. Second, because over-redaction is survivable, you can afford a deliberately permissive first stage and let a cheaper verifier demote the obvious false positives afterwards. The failure you cannot recover from is the one where nothing looked at the span at all.

entities:
  CREDIT_CARD:   {action: block,      strict: true,  checksum: luhn}
  NATIONAL_ID:   {action: block,      strict: true,  checksum: none}
  EMAIL:         {action: tokenize,   strict: true,  scope: session}
  PHONE:         {action: mask,       strict: false}
  PERSON:        {action: mask,       strict: false}
  ORG:           {action: allow}      # too many false positives to redact
# strict: an ambiguous span is redacted by default rather than passed through

Keep the cost of over-redaction visible rather than dismissing it. An assistant that turns "reset the password for jane@acme.com" into "reset the password for <EMAIL>" with no way back cannot complete the task. That is a product failure, not a security win - which is exactly why the choice of redaction technique matters as much as the detection.

The redaction spectrum: what each technique preserves

"Redaction" covers five materially different operations. The right one is decided by what the downstream consumer needs, not by which is most secure.

Suppression is rarely right: a downstream reader cannot distinguish a redaction from a truncation. Typed placeholders are the sane default for anything sent to a third-party model, and the numbering matters - the same value appearing three times must receive the same index, or the model loses the coreference and answers about three different people. Partial masking is the support-desk standard and is weak against an adversary holding auxiliary data; last four digits plus a name is often enough to re-identify.

Tokenization buys referential integrity: with deterministic tokenization the same input always yields the same surrogate, which is what lets two redacted datasets be joined. That determinism is also a leak - anyone who sees the token stream can count distinct individuals and correlate sessions without ever learning a value. Format-preserving encryption (NIST SP 800-38G, FF1 and FF3-1) gets you the same property with a key instead of a lookup table, so there is no vault to replicate or scale, at the cost that key compromise decrypts everything at once.

Synthetic replacement is the underused one and the right choice whenever the redacted text will be read by a model. <PERSON> is out-of-distribution and measurably degrades comprehension; a plausible fabricated name does not. It is the correct technique for building fine-tuning corpora and evaluation data. Its danger is that consumers forget the values are fake and act on them, so mark synthetic records at the record level, not just in a README.

The rule of thumb: human reader gives partial mask; model reader gives synthetic or typed placeholder; a machine that must act on the value gives tokenization with scoped unmasking; analytics that must join gives deterministic tokenization or FPE, accepting the equality leak knowingly.

Reversibility, and where the token vault lives

The moment redaction becomes reversible you have built a system that holds every piece of PII the pipeline has ever seen, in one place, indexed and queryable. The vault is now the highest-value asset in the deployment and has to be designed as one: its own datastore, its own credentials, encryption at rest under a KMS key the application service account cannot use for bulk decryption, per-call authorisation, and an access record for every detokenization written to the audit trail.

Scope tokens as narrowly as the use case allows. Session-scoped mappings that expire with the conversation cover the common case - restoring the user's own values into the response - and bound both the size and the lifetime of the vault. Promote to a long-lived, globally consistent token only when a concrete requirement needs cross-session joins, because that promotion is what turns a transient cache into a permanent PII database.

Who may detokenize is a service-level decision, not a user-level one. The email service needs the address; the analytics job does not; the model never does. Enforce with per-type scopes - a service holding detokenize:EMAIL and nothing else cannot exchange a NATIONAL_ID token even if it obtains one.

Watch the loop that quietly defeats the whole design: a token that round-trips through the model. If the post-processor detokenizes any token-shaped string it finds in the output, an injected instruction can induce the model to emit a token belonging to another session and your own post-processor will decrypt it for the attacker. Only substitute tokens minted for the current request context, and verify ownership before substitution.

Finally, the vault inherits the deletion obligation. Mapping rows without a TTL are the reason a "delete my data" request cannot be satisfied.

Three places an LLM system leaks PII

These are genuinely different problems with different controls, and conflating them is why teams build a detector and still leak.

1. In transit - prompts, context, and logs

By a wide margin the most common real-world leak, and it is not a model behaviour at all - it is ordinary data handling. Retrieval makes it worse: a RAG query can surface another user's or another tenant's record into the prompt, where it presents as a relevance bug rather than as a breach. See tenant isolation for the retrieval-side controls.

2. In the weights - memorization from training data

This applies only if you pre-train or fine-tune on data containing PII; it does not apply to calling a hosted model. The levers are deduplicate the corpus (a record seen many times is memorized far more strongly), redact or synthesise before the corpus is built rather than after, and measure with planted canaries. The measurement machinery - the confidence gap, DP-SGD, running the attack as an audit - is covered in membership-inference defense and is not re-derived here.

3. In generated output

The model emits PII that was not in the user's prompt: recalled from context that should not have been assembled, retrieved from the wrong document, or simply hallucinated. A fabricated but well-formed phone number is a support problem rather than a privacy breach; a hallucinated number that happens to belong to a real person is luck, not design. Output scanning is the control, and the network-side half of it - destinations, canaries, markdown image exfiltration - lives in egress filtering.

Logging prompts and responses without building the breach

If a PII programme fails in production, this is usually where. Redaction happens in the request path; logging happens everywhere else, written by teams who never saw the privacy design. You cannot redact a sink you have not enumerated, so enumerate first:

application logs; APM and distributed-trace span attributes carrying prompt text; exception trackers that attach the request body or serialise local variables into a stack frame; the model provider's own retention window, which is typically measured in days and needs a zero-retention arrangement to eliminate; response caches, including semantic caches keyed on the prompt text itself; the evaluation and annotation pipeline, where production traffic is sampled into a dataset that engineers browse; product analytics events; crash dumps and core files.

Redact at the boundary, before the loggable object exists. If the log record is constructed from the raw request and a downstream filter is expected to scrub it, you are one misconfigured appender away from raw PII on disk. The reliable pattern is that raw text lives in exactly one variable for the shortest possible time and every persisted representation is derived from the redacted copy.

Where you still need linkability - "did this incident involve that customer" - store a keyed HMAC of the identifier rather than the identifier. It answers the investigator's question, the log never holds the value, and rotating the key destroys the linkage on demand.

Two things that are not controls. Sampling is not a control: one percent of PII-bearing logs is still PII-bearing logs, just harder to find. Reviewing the code is not a test. The test is to plant a unique canary identifier through a load test and then search every sink for it. The sinks it turns up in are your actual exposure; the ones you designed for are merely the ones you thought of.

Retention, deletion, and the fine-tuned model problem

Retention has to be enforced by the store, not by a cleanup job. Object lifecycle rules, table TTLs and index expiry survive an on-call rotation; a cron job that can be paused will eventually be paused and nobody will notice for a year.

A deletion request covers the easy ninety percent mechanically: chat history, vault mappings, cached responses, the evaluation sample, the analytics warehouse copy, the search index. Backups are the first genuinely hard component, because you cannot rewrite an immutable backup. The defensible position is documented rather than clever: backups are retained for a bounded period, deleted subjects are held on a suppression list applied after any restore, and the record is therefore gone within that bounded period. Regulators accept a bounded window; what they do not accept is an unbounded one.

The record that is inside the weights

If the individual's data was in a fine-tuning corpus, you cannot delete a row from a matrix. The options, from least to most defensible:

Argue the weights are not personal data. Fragile. Authorities have shown willingness to order the deletion of models trained on unlawfully processed data, so this is a position, not a plan.

Machine unlearning. Gradient-based methods that attempt to undo a specific example's contribution are an active research area with no accepted way to verify the influence is actually gone - and verifiability is the entire point of a deletion promise. Name it, and do not build a compliance commitment on it today.

Output-level suppression. Filter generations for the deleted individual's identifiers. This is a real mitigation and should be switched on immediately on receipt of a request, but it is suppression, not erasure, and must be described as such.

Retrain from a checkpoint on the corpus with the record removed. This is the answer that actually works, and it imposes one architectural requirement worth more than all the others: the fine-tuning corpus must be regenerable - derived from a source of record by a versioned, reproducible transform, so that deleting at source propagates on the next build. If the corpus is a hand-curated artefact that nobody can rebuild, you cannot honour deletion at all, and that fact is discovered at the worst possible moment.

The commitment you make is therefore a cadence, not an event: deletion requests are reflected in the served model at the next scheduled retrain, which happens at most every N weeks. N is a number you choose deliberately, can defend, and which bounds worst-case honouring latency. In the window between request and retrain, output suppression carries the residual risk, and you write that down.

The cheapest version of this whole problem is to not have it. Synthetic replacement at corpus-build time keeps identifiable values out of the weights in the first place, and costs a fraction of solving it afterwards.

Evaluating a PII pipeline: labelled sets, per-entity recall, residual risk

Vendor-published accuracy is measured on the vendor's distribution. Your traffic is lowercase, multilingual, full of product codes and pasted log fragments. The only number that means anything is the one you measured on your own data.

Build a labelled set from real traffic. Sample production prompts, have humans annotate character-offset spans with entity types. Several hundred to a couple of thousand examples is enough to be decision-useful. The annotation cost is genuine and is precisely why most teams skip this step, which is why most teams do not know their recall. Double-annotate a subset to measure agreement: where two trained annotators disagree about what counts as PII, the finding is that your policy is underspecified, not that the annotators are careless.

Measure per entity type, always. An aggregate recall figure is dominated by whatever entity is most frequent - usually email or person - and will cheerfully hide that national-ID recall sits at 0.6. Report a table, not a number.

Decide how you score partial spans. If the detector caught the surname of a full name, exact-match scoring counts that as both a miss and a false positive, while operationally it is a partial leak - which is a leak. Score strictly for anything you make a safety claim about, and track partial matches separately as their own failure class, because they behave differently from complete misses when you go to fix them.

Report precision alongside recall so the over-redaction cost stays visible as a product metric rather than becoming an invisible tax, and track p99 latency, because the detector sits in the request path and a synchronous model call there changes the shape of the whole service.

Augment synthetically for rare entities. Real traffic will never contain enough passport numbers to measure recall on them. Generate format-valid examples embedded in realistic sentences, keep them clearly separated from the real-traffic set, and read synthetic recall as an upper bound rather than an estimate.

Gate releases on it. Entity lists, model versions and pattern sets all change; every change reruns the eval and a per-entity recall regression blocks the deploy. Without a gate, the pipeline silently degrades and the first evidence is an incident.

The residual-risk conversation

Write down the measured numbers and what they imply in plain language. "Person-name recall on lowercase chat input is 0.91" means roughly one name in eleven passes through untouched. That sentence is the single most valuable output of the whole evaluation effort, because stakeholders who believe the pipeline is a guarantee will approve things on the strength of "we have PII redaction" - sending traffic to a third-party model, widening a data-sharing agreement, loosening an access control.

Give them the number, state the compensating controls that cover the gap (contractual zero-retention, tenant isolation, output scanning, bounded retention), and record the acceptance with a named owner. A measured and disclosed 0.91 is a defensible engineering position. An unmeasured "we redact PII" is not a position at all.

End-to-end PII handling flow

Trace one request through the whole stack. A support agent types: "Send an email to john.reid@example.com confirming the refund on card 4111 1111 1111 1111."

Detect. The pattern layer matches the email and the sixteen-digit run; the run passes Luhn, so it is promoted from CARD_CANDIDATE to CREDIT_CARD at high confidence. The NER layer tags "John Reid" as a person from the local part and from the surrounding sentence.

Policy. CREDIT_CARD is configured to block outright - no downstream service in this deployment has a legitimate need for a full PAN, so the request is refused with an explanation rather than silently mangled. EMAIL is configured to tokenize, because the email tool genuinely needs the value.

Redact and call. The model sees "Send an email to <EMAIL_a1b2> confirming the refund." The mapping a1b2 -> john.reid@example.com is written to the vault scoped to this session with a short TTL.

Act. The email service holds the detokenize:EMAIL scope, exchanges the token for the address, and sends. That exchange is recorded: which service, which token, which request correlation id, at what time.

Persist. Every log record - application log, trace span, eval sample - is derived from the redacted copy, never from the original buffer. An HMAC of the address is stored alongside so an investigator can later confirm which customer a record concerned without the log holding the address.

Erase. A deletion request three months later drops the vault rows and the chat history immediately, applies a suppression entry so a backup restore does not resurrect them, and - because this deployment fine-tunes on redacted transcripts - is reflected in the served model at the next scheduled retrain.