Dead-letter review: quarantine and redrive

Inspect quarantine separately from effects

Four independent messages each intend one counter increment. Healthy succeeds immediately. Transient fails its first two deliveries before any effect. Poison always fails schema handling until the operator repairs the handler. Uncertain commits successfully but loses its acknowledgement until the acknowledgement channel is repaired. This local deterministic experiment maintains a source schedule, a quarantine state and an effect ledger. It does not connect to a broker or implement an ADK API. In the default first pass, two effects have happened but three messages enter quarantine. A dead-letter entry therefore does not prove that its operation never committed.

Choose an attempt budget and retry interval

Maximum receives per cycle includes the initial delivery, not just retries. With two receives, transient is quarantined; with three, its third delivery succeeds in the source cycle. Retry interval spaces failed deliveries by a fixed number of modeled seconds. Healthy completes independently of poison; this fixture does not preserve strict FIFO business ordering. Changing any parameter rewinds and pauses the experiment. Next event advances one transition, Play advances one transition per three inspection seconds, Pause freezes it, Restart retains parameters and speed, and Time scrubs to a visible snapshot. Speed controls inspection pace only. Modeled queue time, printed in the diagram and trace, is separate from playback duration.

Recover deliberately after diagnosis

Operator action chooses hold all, redrive all retained messages, redrive only poison, or redrive only uncertain. Redrive occurs once after the first source cycle drains and the Operator wait elapses. Repair before redrive fixes the poison handler and the lost acknowledgement channel; it does not edit the business payload. Redrive resets the cycle receive count while retaining the total receive count and application intent. Default repair, stable keys and sixty-second result retention lead to four acknowledged messages and exactly four increments. Clear Repair before redrive: poison and uncertain exhaust their new cycle and return to quarantine. Transient can still recover because its outage is limited to the first two deliveries. There is no infinite redrive loop.

Preserve application identity across redelivery

Deduplicate committed intent keys stores a result when an effect commits, with expiry measured from that commit. A repeated delivery before expiry returns the result without another increment; it does not extend expiry. Stable redrive identity preserves the original application key. New identity assigns a different key only on redrive, so uncertain commits again and the default total rises to five. Disable deduplication and the two initial uncertain deliveries plus its repaired redrive produce three effects for that intent, six effects overall. Result retention of zero also permits duplicates. A retained result is valid only strictly before expiry: arrivals at the exact expiry time execute again. This is an atomic in-memory fixture, with no storage failures, concurrent consumers, key-payload conflicts or exactly-once claim.

Distinguish message retention from result retention

Quarantine retention starts when each message enters quarantine. At the operator inspection, a message whose age is at least the retention value expires and cannot be redriven. Use wait sixty and retention sixty: the three default quarantined messages expire, leaving just the original two effects. This synthetic policy deliberately uses time since quarantine; real brokers can use different timestamp and retention rules. Message expiry never rolls back an effect. Result retention belongs to the deduplication store and is independent: wait sixty with quarantine retention one hundred twenty retains messages but expires uncertain’s stored result, so a stable-key redrive can duplicate its effect. Choose timeouts and retention for a real system using its actual contracts.

Read and export the evidence

Current snapshot lists each message, reason, key, cycle, receive count, effects and deduplication hits. Recent transitions describe the same state as the diagram; horizontally scroll the diagram on a phone using touch or arrow keys. Export queue JSON saves the entire synthetic run. Capture baseline saves the visible caption; Replace baseline updates it and Clear baseline removes it. Notebook observations persist locally and Export notes downloads them. Expand experiment and Exit full screen change the view. Playback pauses when the tab is hidden; parameters never start playback automatically. The fixture excludes distributed delivery races, permissions, visibility leases, acknowledgement latency, exponential backoff, real storage durability and broker-specific redrive quotas. Its purpose is to make quarantine, recovery identity and previously committed effects inspectable.