FAULT INJECTION · REQUEST AND EFFECT INSPECTOR
Fault injection: lose an acknowledgement after a side effect
Inject a lost response after the server commits, then follow retries, deduplication and the committed-effect ledger.
Scroll the diagram horizontally to inspect every label. The panels below give the same state as text.
Your experiment
Keep the lost first response and clear server deduplication. Compare the client outcome with the committed counter: the client reports one success while the ledger shows two increments.
Then restore deduplication and set result retention to zero, or issue a new key on every attempt, and explain why each change breaks idempotency in a different place.
Recent events
Offline application-policy model, not an ADK retry contract, a live API or a fault-injection tool. One client intends one counter increment. Server operations continue after client timeouts. This server atomically reserves an in-flight key, then retains its result for the configured duration after commit. Response transit is fixed at 50ms; completed responses at an exact deadline are handled before timeout. Backoff doubles up to eight seconds; optional jitter uses a fixed illustrative sequence. A 400 is non-retryable here; a first 503 is retryable. No crashes, durable storage, key-payload conflicts or multi-client retry storms are modeled. Idempotency reference · Timeout and retry reference.
Inject the fault after the side effect
The fault in this experiment happens at the worst moment: the server has already committed the increment, and the success response is lost on the way back. The client cannot tell that apart from a request that never arrived. With the opening settings the server needs two seconds, the client timeout is three seconds and the first success response is lost. Attempt 1 times out at three seconds even though the server committed at two. The client retries after a one-second delay.
Deduplication turns the retry into a lookup
Server deduplicates operation keys is enabled and the retry reuses the logical intent key, so the second attempt finds the stored result. It is acknowledged at 4.05 seconds, the committed counter is one, and one deduplication hit is recorded. Clear deduplication: the retry executes again, is acknowledged at 6.05 seconds, and the counter is two. The client reports success in both runs; only the effect ledger shows the duplicate.
Break idempotency in two other ways
Restore deduplication and choose New key for every attempt in Retry key. The server cannot recognize the repeated intent and commits twice. Restore the stable key and set Stored-result retention to zero: the stored result is forgotten before the retry arrives, and the counter is again two. Choose Lose all success responses with deduplication on: three attempts time out, the budget is exhausted at twelve seconds and the intent is retained for reconciliation, yet the counter is one. Without deduplication the same run commits three increments.
Other injected faults and timing
First attempt gets 503 fails before any commit and the retry succeeds with one increment. Reject input with 400 is non-retryable and ends rejected with zero increments. No response fault succeeds at 2.05 seconds. Timeout per attempt below Server execution time makes retries overlap work still in flight: with a one-second timeout the retry finds the reserved key and is acknowledged at 2.05 seconds. Maximum total attempts, Initial retry delay and Deterministic full-jitter example change the retry schedule.
Inspection and limits
Play, Next event, Restart, Speed and Time step through attempts, server operations and the committed-effect ledger; Recent events lists every transition. The opening replay lasts 4.05 model seconds, so the timeline stretches it across sixty inspection seconds; the time readout shows both clocks and every event keeps its model time. One client and one counter are modeled with fixed fifty-millisecond response transit. There are no server crashes, durable deduplication storage, conflicting payloads under one key, multi-client retry storms or real fault-injection tooling. In real tests the injected fault must be scoped and reversible, and the assertion must check side effects, not only client status.