A credential-leak score depends on both the response and the detector that reads it. Expanding a recognition pattern, changing a boundary condition or widening an exclusion can move the score even when the response stays identical. A useful evaluation makes those inputs explicit and uses synthetic fixtures to distinguish detector coverage from agent behavior.
Define the matching contract
For each supported format, specify what text should match, which surrounding characters are allowed and which part of a match identifies the candidate. Keep the pattern set and exclusions versioned with the corpus. Include deliberately invalid examples that resemble the target format: matching every token-like string produces a noisy score with little diagnostic value.
Separate detection from investigation logs. A detector may need candidate text, while an ordinary event log generally needs only a restricted evidence reference, detector version and outcome. OWASP's logging guidance and CWE-532 describe risks from sensitive information in logs. Design that evidence path before collecting evaluation results.
Query order should not decide whether a fixture matches
A signed storage URL puts credential-bearing information in query parameters. An application or URL builder may arrange those parameters differently. If a detector recognizes only one ordering, an ordinary serialization change can alter its apparent leak rate without changing the underlying exposure.
Consider a synthetic signed-URL fixture with four distinct parameters. There are 24 possible orders. Running every order against two detector configurations gives 48 combinations. This is test-matrix arithmetic rather than a measured detector result. The purpose is to avoid treating one hand-written URL as adequate coverage of a structured format.
Exclusions need an exact fixture identity
Red-team evaluations often need deliberate fake credentials. Suppose an original test fixture contains a synthetic signed URL and the harness is told to exclude that exact value. The same bytes appearing in the response should not count as evidence that the agent discovered a new secret. But a changed permission parameter should remain a distinct candidate match.
Exercise this distinction by changing a synthetic permission value and checking exclusion behavior. An exclusion defined only by a friendly resource name or a broad prefix could hide more than the intended fixture. Store an explicit fixture identifier and its permitted matched representation. Review changes to exclusions as changes to the evaluation, since they alter what the score means.
Recognizing a shape does not establish a live credential
A regular expression finds text shaped like a token. It does not establish that the credential is valid, that its permissions are dangerous or that the model retrieved it from a protected source. A score can also miss transformed text, split values or a format outside its vocabulary. Treat these as limits of the detection method, not as proof that a passing response contains no sensitive information.
Keep the evaluation entirely synthetic. Do not send real secrets to a model to test whether they come back, and do not call a provider's authentication endpoint just to check whether a matched candidate works. Instead, record the detector version, fixture provenance, selected pattern set and exclusion set alongside each result. Compare runs only when those inputs are known.
Use a matrix rather than one happy-path example
An original acceptance plan has four axes: positive versus negative shape, plain versus surrounding text, ordinary versus reordered parameters, and excluded versus changed fixture. Add length boundaries and non-ASCII surrounding text where your users need them. A helpful negative fixture resembles a token enough to exercise a boundary but is explicitly invalid by the detector's intended format.
Review detection and response separately. The scorer can label a candidate leak; your application still has to decide what to redact, whether to retain the evidence and who may inspect it. Avoid placing matched token text in general logs. Keep restricted evidence references so an analyst can investigate without copying the sensitive value into every monitoring system.
Interpret score changes before declaring a regression
After an upgrade, a higher leak count may indicate broader recognition rather than a less safe agent. Re-score a fixed synthetic response corpus with the old and new detector revisions, then inspect which cases changed. Separate newly recognized formats from changes in generated behavior. Record the exact responses, detector configuration and acceptance rules so a future comparison can explain which boundary changed.
Sources and evidence
Version detectors and exclusions with the evaluation corpus. Parameter-order and boundary tests reduce format blind spots, while a regex match remains evidence of token-shaped text rather than proof of a valid credential.