Research review / research

Anthropic's Agent Memory Tests: Letting Claude Decide Beat Rules, and a Wiki Helped Only Sometimes

A November 6 Anthropic post breaks agent memory into note-taking and offline 'dreaming' passes. Telling strong models to record only failures cut their gains, one model's nine-file memory outperformed another's 413 files, and a wiki built from a 420M-token corpus added 8 points over search alone.

By Sandeep Belgavi · Published 2026-11-07

Anthropic published "The components of agent memory" on November 6, 2026. It defines memory narrowly as knowledge an agent gains from experience and keeps between sessions as readable context, and it excludes weight changes and compaction. Its central idea is a split between notes written during work and a later offline pass, which it calls dreaming, in which the agent reviews logs to correct mistakes and reorganise what it knows. Anthropic credits the sleep-time compute idea to researchers at Letta and UC Berkeley.

What the experiments showed

  • Judgment beat fixed rules for stronger models. On CL-Bench 1.0, a rule to "record only failures" removed much of the benefit of memory for Opus 4.7 and Opus 5. Older models were the opposite: prompting Sonnet 4.6 to read its memory and test its theories nearly doubled its score.
  • Less was more in Pokemon. About 10,000 steps into one run each, Sonnet 3.7 had written 413 memory files across 225 folders and earned one badge. Fable 5 had written nine files in two folders plus a top-level global.md and earned four. The useful notes were the ones the game did not already display.
  • Dreaming caught a real error. One note placed a landmark on the wrong side of a mountain. A dreaming pass checked it against the transcript and fixed it. With the corrected note the run reached the next city, and with the uncorrected note it never did.
  • Wikis have a narrow payoff. On about 350,000 files and roughly 420M tokens from Stainless, a company Anthropic acquired in May, 128 dreaming sessions lifted a wiki-only score from 13 percent to 48 percent of graded criteria. Search plus the wiki reached 61 percent, against 53 percent for search alone, and saved about 20,000 tokens per question.

Where it did not help

On 24 merger questions, many needing several documents, Claude scored 87 percent with the wiki on those multi-document items against 73 percent with search alone. On a legal-brief task where most criteria could be met from one document, the wiki showed no clear gain. On another task the wiki grew larger than the source files, which Anthropic compares to the map as large as the empire in Borges.

The most practical failure came from concurrency. Several agents writing to one shared table of contract terms overwrote each other, and only 41 of 57 graded facts survived. Giving each contract its own page, written by one agent and quoting terms word for word, reached 57 of 57.

What remains uncertain. These are Anthropic's own evaluations of Anthropic's models, some on a single run or a handful of questions, and the post does not publish variance for most comparisons. Dreaming also costs compute, and Anthropic says it is still unclear how often to run it. A separate November 2 paper, EvoMemState, argues that memory kept as independent facts misses how new facts update or conflict with old ones, which suggests the representation question is open.

Analysis: Copy the discipline before the architecture. Keep the always-loaded layer small, verify notes against a primary record, and give concurrent writers separate files. If Geetha's help-desk agent at Belgavi's Book Store remembers that a publisher's returns window is 30 days, that note should cite the CASE ticket it came from, so a later review can confirm it and change it when the policy changes.

Sources and evidence

Anthropic's November 6 memory study found fixed note-taking rules hurt stronger models, small curated memory beat exhaustive logging, offline dreaming corrected a wrong note against the transcript, and a wiki added 8 points over search on one corpus but nothing on single-document tasks. Concurrent writers to one table lost 16 of 57 facts until each contract got its own page. All results are Anthropic's own.

Views are my own and do not represent any employer, past or present.