A payment API that is safe to call twice is a different product from one that is not. Every network between an agent and a payment rail delivers at least once: when a request times out, the client cannot tell whether the request was lost on the way in or the response was lost on the way back. The safe reaction — retry — is also the one that double-charges a naive server. Idempotency keys convert an at-least-once delivery guarantee into an exactly-once effect, and they matter most in agentic payments, because an autonomous agent retries on a policy rather than on a human’s hesitation. Here are the contract, the server-side machinery, the replay rules, and the one duplicate-payment hazard idempotency cannot catch.
At-least-once is a choice, not a bug
It is tempting to read a timeout as a failure. It is not — it is an ambiguity. When an agent POSTs a charge and the socket goes quiet, one of two histories is true: the request never landed, or it landed, executed, moved money, and the acknowledgement died on the return path. Nothing the client can observe distinguishes them.
Distributed systems resolve that ambiguity by choosing a delivery semantic. At-most-once (never retry) loses transactions: legitimate payments silently vanish, leaving abandoned carts and held inventory. At-least-once (always retry) never loses one, but duplicates freely. Every serious payment stack and message bus picks at-least-once, because losing money movement is worse than repeating it — provided you can collapse the repeats.
Idempotency is how you collapse them. It does not make the transport deliver once; it makes the effect happen once however many times the transport delivers.
The contract: one key per logical operation
An idempotency key is an opaque string that names one logical operation, not one HTTP attempt. Every retry of that operation carries the same key; a genuinely different payment carries a different one. The server promises that for a given key it performs the effect at most once and returns a consistent answer to every later request bearing that key.
The client mints the key, and that placement is deliberate: only the client knows which attempts are “the same operation.” The server sees two byte-identical requests and cannot tell a retry from a customer legitimately buying the same coffee twice.
The key travels on the mutating request, conventionally as an Idempotency-Key header, and must cover the whole effect the caller cares about. If one call authorizes a payment, writes a ledger entry, and emits a webhook, the key protects all three or none.
Derive the key from the request, never per attempt
The commonest implementation bug is generating the key in the wrong place. A random UUID minted inside the retry loop is worse than useless: every attempt carries a fresh key, every key misses the store, and the idempotency layer faithfully executes each duplicate while the dashboard shows the feature “enabled.”
The key must be created once, when the operation is decided, and reused by every attempt — including attempts after a process restart. Two patterns work. Mint a random key when the payment record is first persisted, so any worker picking the job up reads the same key from durable state. Or, stronger for agents, derive the key deterministically: a hash over the stable identity of the intent — mandate identifier, payee, amount, currency, and a durable caller-side operation id. Derivation needs no shared memory: two agent instances that independently reconstruct the same intent compute the same key, and their requests collide on the server exactly as intended.
Server side: the atomic claim
On the server a key is a row, and the operation that matters is an atomic insert-if-absent. Before any money moves, the handler creates a record for the key with status in progress. If the insert succeeds, this caller owns the operation and proceeds. If it fails on a uniqueness violation, the operation already exists and this request must execute nothing.
The atomicity is the whole point. A read-then-write — look the key up, execute if absent — leaves a window in which two requests both read absent and both charge. Only a conditional write enforced by the store itself (a unique constraint, a compare-and-set, a SET NX) closes it.
The record must be durable and strongly consistent: it is now the system of record for “did this happen.” A cache that can evict a key or serve a stale miss reopens the double-charge window. On completion the record is updated to a terminal status with the response produced.
Concurrent duplicates and the in-progress state
The interesting case is not the retry a minute later but the one arriving while the original is still running. Agents produce these constantly: an aggressive client timeout fires long before a payment rail has finished.
The loser of the atomic claim finds a record whose status is in progress: the effect is neither confirmed nor ruled out, and there is no stored response to replay yet. Two honest answers exist. The server can hold the request briefly and return the outcome once the winner records it — pleasant for clients, but it ties up a connection. Or it can return immediately with a conflict status meaning in progress, retry shortly, commonly 409, which the client must treat as retryable, not fatal.
What it must never do is fall through and execute, or report the payment as failed: at that instant nobody knows, and a false negative is how an agent gets talked into paying again.
Replay the original response, not a fresh one
When a key hits a completed record, the server returns the stored response — the same status code and body the first attempt produced — rather than recomputing an answer from current state. That is subtler than it sounds: if the server re-reads the payment and re-serializes it, a retry landing after the payment was captured, refunded, or disputed returns a different object than the original call did, and the client silently observes a world it never asked about.
Replay applies to failures too. If the first attempt was declined, the retry returns that decline — not a fresh authorization that might now succeed. Marking replays with a response header is invaluable in debugging.
One guard belongs here: store a fingerprint of the request payload alongside the key. If the same key arrives with a different amount or payee, that is a client bug, not a retry, and the answer is a conflict error rather than a misleading replay.
Scope and lifetime: what a key means and for how long
A key is never global. It is scoped to the credential that presented it — merchant, API key, or tenant — so unrelated callers cannot collide and none can probe another’s history by guessing keys. Many implementations scope by endpoint too, so the same key on a different operation is a conflict rather than a confusing replay.
Lifetime is an operational decision with a hard consequence. Keys are retained for a bounded window — twenty-four hours is a widespread choice — then garbage-collected. That window is not storage hygiene: it is exactly the period during which a retry is safe. Once the record expires the key is a stranger again, and retrying executes a brand-new payment under it.
Clients must therefore treat retention as a deadline: retry inside it freely, and past it stop retrying and resolve the outcome by querying the payment, because the safety net has been taken down.
The agent hazard idempotency cannot catch
Idempotency deduplicates by key. It says nothing about two requests that are semantically the same payment but carry different keys — exactly what a re-planning agent produces.
The agent attempts a payment, hits a timeout, and instead of retrying the same call it re-enters its planning loop, concludes afresh that the user still needs the item, builds a new request, mints a new key, and pays again. The same happens when a crashed agent restarts from a checkpoint predating the payment, or when two sub-agents are handed the same task. Every layer behaves correctly; the customer is charged twice.
The fix is the derivation rule applied at the level of intent: make the key a deterministic function of the mandate — the signed statement of user intent the payment fulfils — not of the attempt. Hash the cart mandate identifier with payee and amount, and an agent that reconstructs the intent reconstructs the key. Back it with mandate-level accounting server-side: track how much has been drawn against a mandate and refuse a second full charge under it.
Reconciliation as the backstop
Some ambiguity survives every layer above. The classic case: the server claims the key, calls the rail, the charge succeeds, and the process dies before writing the terminal status. The record sits in progress forever, and no local retry resolves it — the truth lives at the rail.
The backstop is a sweeper that finds records stuck in progress past a threshold and asks the rail what happened, using the same key propagated downstream as the rail’s reference so the query is answerable. What it learns becomes the terminal state, which later retries replay.
A useful companion is a duplicate-charge alert: flag repeat charges with the same payer, payee, amount, and currency in a short interval even under different keys. It is heuristic, so it belongs in review rather than a hard block — but it is how you catch a client minting keys wrong.
A worked failure timeline
T+0s: the agent derives key k from the cart mandate and POSTs. T+0.1s: the server claims k as in progress and calls the rail. T+5s: the agent’s client timeout fires; it retries with k. T+5.1s: the claim fails, status is in progress, the server returns conflict – retry shortly. T+8s: the rail approves; the server records success and the response body. T+10s: the agent retries again; the stored response is replayed. One charge, three requests, and a client that never had to reason about it.
Now break it. Replace T+8s with a crash: the rail approved, the server died before recording it. Every retry finds k in progress and gets a conflict until the client hits its ceiling and escalates. Minutes later the sweeper queries the rail for k, learns the charge settled, and writes success. The customer was charged once. That is the whole promise: not that failures stop, but that every failure lands on a key whose recorded state eventually says what did and did not occur.