Context packing architecture — salience, summarization, working memory

The core tension: a fixed window, unbounded context

Every context window is a fixed token budget shared by everything the model needs to see: the system instructions, the user’s task, retrieved knowledge, few-shot examples, tool definitions, conversation history, and — crucially — room reserved for the model’s output. The candidate material that could go in is effectively unbounded and grows every turn.

Packing is therefore an optimization problem: maximize the relevance of what you include subject to a hard token constraint. More is not better — every token you spend on a marginally-relevant chunk is a token unavailable for something better, and (as the failure-mode section shows) irrelevant context actively degrades output. The goal is the highest-signal subset, compressed and ordered, not the largest one.

Anatomy of a context pack

A context pack is the assembled prompt, and it is useful to think of it as distinct categories competing for budget, each with different volatility and value:

Naming the categories is not bureaucracy — the stable ones can be cached and placed up front, the per-query ones must be selected fresh, and the budget must be split across them deliberately rather than letting whichever retriever ran last consume the whole window.

Salience scoring: what earns a place

Salience is how relevant a candidate chunk is to the current task — the score that decides what makes the cut. Naive systems use a single signal: cosine similarity between the query embedding and chunk embeddings, take top-k. That is a floor, not a ceiling.

Stronger salience is hybrid and multi-signal. Combine dense (embedding) retrieval with sparse lexical retrieval (BM25) so exact terms and semantic matches both surface. Then rerank the merged candidates with a cross-encoder, which scores query and chunk together and is far more accurate than the bi-encoder used for first-pass retrieval. Layer in task-specific signals: recency (newer docs weighted up), authority or source trust, and explicit metadata filters. Finally, apply diversity — maximal marginal relevance (MMR) — so you don’t spend the budget on five near-duplicate chunks that each say the same thing.

The output of this stage is a ranked, de-duplicated candidate list with scores — the raw material the budget stage then has to fit.

Token budgeting as constraint satisfaction

With ranked candidates in hand, budgeting decides how much of each category actually goes in. Treat it as explicit allocation, not first-come-first-served. Reserve tokens for the model’s output first (a truncated answer is the worst outcome). Reserve fixed budgets for the stable categories — system, tools, exemplars. Then allocate the remainder across retrieved knowledge, working memory, and long-term memory according to the task.

The allocation should adapt: a factual lookup deserves more retrieved-knowledge budget; a long multi-turn debugging session deserves more working-memory budget. When a category’s ranked candidates exceed its budget, you don’t simply drop the tail — you compress it, which is the next stage. Budgeting and summarization are two halves of one decision: given N tokens for this category and M tokens of good material, fit M into N with the least information loss.

Summarization and compression

When good material exceeds its budget, compress rather than truncate. Several techniques, increasingly aggressive:

Extractive compression pulls the most salient sentences or passages out of a long chunk and drops the rest — cheap, lossless within what it keeps, and preserves exact wording (important for quotes, code, numbers). Abstractive summarization uses a model to rewrite a body of text into a shorter form — higher compression, but it can drop or distort details, so it suits background context more than load-bearing facts. Hierarchical summarization builds summaries of summaries, letting you fold a large corpus into a small pack while retaining a drill-down path. For conversations, a rolling summary condenses older turns into a running digest while keeping the most recent turns verbatim.

The rule of thumb: keep anything the model must reproduce exactly (identifiers, code, figures) in extractive/verbatim form; summarize the surrounding narrative. Compression trades fidelity for budget, so spend fidelity where errors are cheap.

Working memory vs long-term memory

Conversation state splits into two tiers with different packing rules. Working memory is the recent turns kept verbatim — the model needs the exact wording of what was just said to stay coherent. It is small, high-value, and always included. Long-term memory is everything older, which cannot all stay verbatim; it is distilled into durable facts (‘user prefers metric units’, ‘the project uses Postgres’) and stored externally, then recalled on demand when a turn makes it relevant.

The boundary between them is an eviction policy: as the conversation grows, the oldest working-memory turns are summarized into the rolling digest and their facts promoted to long-term store, freeing budget for new turns. This is the same insight as an OS memory hierarchy — a small fast verbatim tier backed by a large compressed recalled tier — applied to a context window.

Priority ordering and the lost-in-the-middle effect

Where you place a chunk matters, not just whether you include it. Models exhibit a well-documented positional bias: they attend most reliably to content at the beginning and end of the context and are most likely to overlook material buried in the middle — the ‘lost in the middle’ effect.

So ordering is part of packing. Put the most important, most task-critical context where attention is strongest: the task/query and the single most salient retrieved facts near the edges, lower-salience supporting material in the middle. Keep stable, cacheable content (system, tools) as a fixed prefix at the very top. The practical consequence is that a pack with the same content in a bad order can measurably underperform one ordered by salience — ranking earns a place, ordering earns attention.

Assembly: delimiters, provenance, and dedup

The final step serializes the selected, compressed, ordered pieces into a single prompt string — and small assembly choices have outsized effects. Use clear, consistent delimiters (headings, tags, or fenced blocks) so the model can tell instructions from retrieved data from history; blurred boundaries are a leading cause of the model treating retrieved text as a command. Attach provenance — source ids or titles — so the model can cite and so you can trace which chunk drove an answer. Deduplicate across categories: the same fact retrieved twice, or present in both memory and RAG, wastes budget and can bias the model by repetition.

Assembly should be deterministic — the same inputs produce the same pack — both for debuggability and for the caching that comes next.

Caching the stable prefix

Notice that the categories split cleanly into stable (system, tools, exemplars) and volatile (query, retrieved chunks, latest turn). If you place all the stable content as a fixed prefix and keep it byte-identical across calls, prompt-caching mechanisms — offered by major model providers including Anthropic’s Claude — can reuse the already-processed prefix instead of reprocessing it every turn, cutting latency and cost substantially on long, repetitive contexts.

This makes ordering and determinism not just quality concerns but efficiency ones: a stable, front-loaded prefix is both where the model attends best and what the cache can reuse. The volatile, per-query material goes after the cached prefix. Designing your pack so the boundary between cacheable and per-turn content is clean is one of the highest-leverage things you can do for a production system’s cost profile.

Failure modes: stuffing, distractors, and injection

Naive packing fails in recognizable ways. Context stuffing — cramming the window full because you can — dilutes attention across too much material and often lowers accuracy versus a tighter pack; more context is not more intelligence. Distractors are retrieved chunks that are topically near but actually irrelevant or contradictory; even a few can pull the model toward a wrong answer, which is why reranking and a relevance threshold (drop low-salience chunks entirely) matter more than raising k. Stale context — outdated memory or documents — produces confidently wrong answers; recency weighting and cache invalidation are the guards.

The most dangerous is prompt injection: retrieved content that contains instructions (‘ignore previous instructions and…’). Because packed RAG text sits in the same window as your real instructions, the model can be induced to follow it. Defenses are structural: strong delimiters and explicit framing that retrieved text is data, not instructions, least-privilege on tools, and treating any retrieved content as untrusted. Packing is thus also a security boundary, not just a relevance one.

Chunking: the upstream decision that shapes everything

Before anything can be retrieved or packed, source documents must be split into chunks — and this upstream choice quietly constrains every stage downstream. Chunks that are too large dilute salience (a chunk is retrieved for one relevant sentence but drags in a page of noise) and waste budget; chunks that are too small fragment meaning across boundaries, so the retriever matches a chunk that no longer contains enough context to be useful.

Good chunking is structure-aware: split on semantic boundaries (headings, paragraphs, function definitions) rather than a blind fixed character count, and add a small overlap so a fact that straddles a boundary survives in at least one chunk. Attach metadata to each chunk at ingest — source, section, timestamp — because the salience and budgeting stages will lean on it later. The reason chunking belongs in a packing discussion is that no amount of clever reranking can recover information that a bad split destroyed: the pack can only be as good as the units it has to choose from.

Query transformation before retrieval

The query the user typed is often not the best query to retrieve with, and transforming it before retrieval materially improves what lands in the pack. Several techniques: query expansion adds synonyms and related terms so lexical retrieval catches more; multi-query generates several paraphrases and unions their results to cover different phrasings; decomposition splits a complex question into sub-questions retrieved separately (essential when one query mixes several information needs); and HyDE (hypothetical document embeddings) generates a hypothetical answer and retrieves against its embedding, which often sits closer to the real answer documents than the terse question does.

All of these feed more and better candidates into salience scoring. The cost is latency and extra model calls, so they suit high-value queries more than every keystroke — but for a hard question, transforming the query is often higher-leverage than tuning the reranker, because it fixes the problem at the source: you can only pack what retrieval surfaced.

Agentic packing: context that evolves across steps

Single-shot RAG packs a context once. An agent packs repeatedly, and its context grows with each step — tool calls, tool results, intermediate reasoning, and observations accumulate turn over turn. Left unmanaged, this hits the window limit fast, and the most valuable early context (the original task, key findings) risks being pushed out or lost in the middle as raw tool output piles up.

Agentic packing adds moves the one-shot case doesn’t need. Tool-result compression: a 10,000-token API response is summarized to the few facts the agent actually needs before it enters the running context. Scratchpad management: intermediate reasoning is pruned or summarized once its conclusion is recorded. Pinned context: the original goal and hard constraints are re-anchored at the edges every step so they never age out. The principle is the same salience-and-budget discipline, but applied dynamically to a context that is a moving target rather than a fixed snapshot.

Evaluating a packer

Packing decisions should be measured, not guessed, and the metrics split across the pipeline. On the retrieval/pack side: context recall (did the pack contain the information needed to answer?) and context precision (what fraction of the packed content was actually relevant, i.e. how many distractors slipped in). On the generation side: faithfulness (is the answer grounded in the packed context rather than hallucinated?) and answer relevance (does it address the query?).

These separate packing failures from model failures. Low context recall means retrieval or budgeting dropped what was needed — fix the pack, not the prompt. High recall but low faithfulness means the model isn’t using what it was given — often an ordering or delimiter problem. Wiring up even coarse versions of these metrics turns packing from folklore into an optimization loop, which is the only way to know whether a change to chunking, reranking, or ordering actually helped.

Structured packing: schemas, tags, and format contracts

How you structure the packed context, not just what goes in it, affects how reliably the model uses it. Wrapping each category in explicit, machine-legible structure — XML-style tags, Markdown headings, or JSON — gives the model unambiguous boundaries: <instructions> here, <retrieved_context> there, <conversation> below. Models follow structured context more consistently than a flat wall of text, and the structure is what lets the model tell an instruction from a document from a prior turn.

The same discipline applies to the output side: a clear format contract — ‘answer as JSON with these fields’, or a specified schema — belongs in the pack and constrains the generation. Structuring retrieved chunks with their provenance inline ([source: doc_42]) additionally makes citations natural and grounds the answer. Structured packing is the bridge between the relevance concerns of the earlier stages and the reliability concerns of production: the better-delimited the pack, the less the model conflates roles, and the harder it is for injected text to masquerade as an instruction.

A reference packing pipeline

Assembled end to end, a robust packer runs as a small pipeline on every turn:

1. RETRIEVE   hybrid (dense + BM25) over the query -> candidate chunks
2. RERANK     cross-encoder + recency/authority -> scored, ranked
3. DIVERSIFY  MMR + dedup -> non-redundant shortlist
4. BUDGET     reserve output/system/tools; allocate the rest per task
5. COMPRESS   extractive for exact facts, summary for narrative
6. ORDER      salient content to the edges; stable prefix on top
7. ASSEMBLE   delimiters + provenance -> deterministic prompt
8. CACHE       stable prefix reused; volatile suffix per turn

Each stage has a clear job, and the failure modes map onto skipped stages: skip reranking and distractors slip in; skip budgeting and one category starves the others; skip ordering and good context gets lost in the middle; skip delimiters and you open an injection seam. The pack, not the model, is where most of these are won.