At-least-once is a choice, not a bug
It is tempting to read a timeout as a failure. It is not — it is an ambiguity. When an agent POSTs a charge and the socket goes quiet, one of two histories is true: the request never landed, or it landed, executed, moved money, and the acknowledgement died on the return path. Nothing the client can observe distinguishes them.
Distributed systems resolve that ambiguity by choosing a delivery semantic. At-most-once (never retry) loses transactions: legitimate payments silently vanish, leaving abandoned carts and held inventory. At-least-once (always retry) never loses one, but duplicates freely. Every serious payment stack and message bus picks at-least-once, because losing money movement is worse than repeating it — provided you can collapse the repeats.
Idempotency is how you collapse them. It does not make the transport deliver once; it makes the effect happen once however many times the transport delivers.
The contract: one key per logical operation
An idempotency key is an opaque string that names one logical operation, not one HTTP attempt. Every retry of that operation carries the same key; a genuinely different payment carries a different one. The server promises that for a given key it performs the effect at most once and returns a consistent answer to every later request bearing that key.
The client mints the key, and that placement is deliberate: only the client knows which attempts are “the same operation.” The server sees two byte-identical requests and cannot tell a retry from a customer legitimately buying the same coffee twice.
The key travels on the mutating request, conventionally as an Idempotency-Key header, and must cover the whole effect the caller cares about. If one call authorizes a payment, writes a ledger entry, and emits a webhook, the key protects all three or none.
Derive the key from the request, never per attempt
The commonest implementation bug is generating the key in the wrong place. A random UUID minted inside the retry loop is worse than useless: every attempt carries a fresh key, every key misses the store, and the idempotency layer faithfully executes each duplicate while the dashboard shows the feature “enabled.”
The key must be created once, when the operation is decided, and reused by every attempt — including attempts after a process restart. Two patterns work. Mint a random key when the payment record is first persisted, so any worker picking the job up reads the same key from durable state. Or, stronger for agents, derive the key deterministically: a hash over the stable identity of the intent — mandate identifier, payee, amount, currency, and a durable caller-side operation id. Derivation needs no shared memory: two agent instances that independently reconstruct the same intent compute the same key, and their requests collide on the server exactly as intended.
Server side: the atomic claim
On the server a key is a row, and the operation that matters is an atomic insert-if-absent. Before any money moves, the handler creates a record for the key with status in progress. If the insert succeeds, this caller owns the operation and proceeds. If it fails on a uniqueness violation, the operation already exists and this request must execute nothing.
The atomicity is the whole point. A read-then-write — look the key up, execute if absent — leaves a window in which two requests both read absent and both charge. Only a conditional write enforced by the store itself (a unique constraint, a compare-and-set, a SET NX) closes it.
The record must be durable and strongly consistent: it is now the system of record for “did this happen.” A cache that can evict a key or serve a stale miss reopens the double-charge window. On completion the record is updated to a terminal status with the response produced.
Concurrent duplicates and the in-progress state
The interesting case is not the retry a minute later but the one arriving while the original is still running. Agents produce these constantly: an aggressive client timeout fires long before a payment rail has finished.
The loser of the atomic claim finds a record whose status is in progress: the effect is neither confirmed nor ruled out, and there is no stored response to replay yet. Two honest answers exist. The server can hold the request briefly and return the outcome once the winner records it — pleasant for clients, but it ties up a connection. Or it can return immediately with a conflict status meaning in progress, retry shortly, commonly 409, which the client must treat as retryable, not fatal.
What it must never do is fall through and execute, or report the payment as failed: at that instant nobody knows, and a false negative is how an agent gets talked into paying again.
Replay the original response, not a fresh one
When a key hits a completed record, the server returns the stored response — the same status code and body the first attempt produced — rather than recomputing an answer from current state. That is subtler than it sounds: if the server re-reads the payment and re-serializes it, a retry landing after the payment was captured, refunded, or disputed returns a different object than the original call did, and the client silently observes a world it never asked about.
Replay applies to failures too. If the first attempt was declined, the retry returns that decline — not a fresh authorization that might now succeed. Marking replays with a response header is invaluable in debugging.
One guard belongs here: store a fingerprint of the request payload alongside the key. If the same key arrives with a different amount or payee, that is a client bug, not a retry, and the answer is a conflict error rather than a misleading replay.
Scope and lifetime: what a key means and for how long
A key is never global. It is scoped to the credential that presented it — merchant, API key, or tenant — so unrelated callers cannot collide and none can probe another’s history by guessing keys. Many implementations scope by endpoint too, so the same key on a different operation is a conflict rather than a confusing replay.
Lifetime is an operational decision with a hard consequence. Keys are retained for a bounded window — twenty-four hours is a widespread choice — then garbage-collected. That window is not storage hygiene: it is exactly the period during which a retry is safe. Once the record expires the key is a stranger again, and retrying executes a brand-new payment under it.
Clients must therefore treat retention as a deadline: retry inside it freely, and past it stop retrying and resolve the outcome by querying the payment, because the safety net has been taken down.