Agent context failure: answer unknown when an early fact is evicted

A lookalike fills the gap

The window holds a fixed 150-token system prompt, the 60-token account fact AC-7731, the later turns and a 40-token final question. With the opening settings, Context budget 1200 tokens, Later turns 12, Tokens per later turn 120 and Evict the oldest turn, the window overflows at turn 10 and the fact leaves together with Turn 2. Turn 8 mentioned another customer, AC-7713, and that turn is still inside. Under Answer with the closest account seen the agent refunds AC-7713: CONFIDENT WRONG ANSWER, with 1150 of 1200 tokens in use. The failure is not size but which content was kept.

Abstaining is the safe failure

Switch When the fact is not in context to Say it is unknown and ask. The verdict becomes ANSWERS UNKNOWN: the agent says it no longer has the account number and asks the user to confirm it. That costs one extra turn instead of a refund to the wrong customer. Back on guessing, set Later turns to 30: the outcome is again ANSWERS UNKNOWN, only because Turn 8 was evicted too. A guessing agent with no lookalike left is lucky, not safe.

Pin, summarize or remember

With 12 later turns and guessing, try each option of When the window is full. Pin the account fact, evict others keeps the fact and evicts Turn 6 instead: CORRECT at 1090 tokens. Summarize evicted turns (80 tokens) keeps AC-7731 inside the summary: CORRECT at 1110 tokens. Write evicted facts to memory, retrieve on demand copies the fact out at eviction, and a lookup returns a 30-token copy before the answer: CORRECT at 1180 tokens. Pinning suits facts known to matter; memory covers unanticipated ones if retrieval finds them.

Find the capacity boundary

With eviction the whole conversation needs 1690 tokens. Raise Context budget to 1700 and everything fits: CORRECT. At 1600 the fact is evicted at turn 13 and the answer is AC-7713 again. Lower Tokens per later turn to 40 and the conversation needs only 730 tokens. Set Later turns to 6 and it needs 970, so the fact survives and Turn 8 never happens. At the 400-token minimum the fact leaves at turn 3; pinning still answers correctly with 370 tokens. A bigger window moves the cliff.

Replay the turns

Any input change recomputes and rewinds; the Complete-run preview shows the outcome at once. Play and Pause run the turns, Next event jumps one turn, Restart rewinds to zero, Speed runs from 0.25x to 2x, and Inspection time seeks from 0 to 60 seconds. The board lists the window after each event. The verdict and metrics appear only at 60 seconds.

Assumptions and limits

Tokens are teaching weights, not tokenizer output. The summary is assumed to keep the account number and the memory lookup always finds it; real summarizers drop identifiers and real retrievers miss. Guessing stands for a model that fills a gap with the closest-looking value in context. Real models can also overlook a fact that is still in the window.