Corrective RAG (CRAG), introduced by Yan et al. in 2024, patches the single most fragile assumption in ordinary retrieval-augmented generation: that whatever the retriever returns is worth reading. Standard RAG stuffs the top-k passages into the prompt and hopes; if the retriever misses, the generator confidently hallucinates on top of irrelevant context. CRAG inserts a small retrieval evaluator between retrieval and generation. It scores how relevant each retrieved document actually is to the query, maps those scores through two thresholds into three verdicts — correct, ambiguous, or incorrect — and runs a different corrective action for each: refine the good documents, fall back to web search for the bad ones, or combine both when unsure. This piece walks the evaluator, the confidence math, the decompose-then-recompose refinement, the web-search fallback, and what the whole loop costs on a CPU-bound small model.
The failure CRAG is built to fix
The quiet flaw in vanilla RAG is that retrieval and generation are wired together with blind trust. The retriever ranks a corpus by embedding similarity, hands back the top k chunks, and the generator treats them as ground truth. But similarity is not relevance: a query about a 2024 policy change can pull back a confident-looking 2019 passage that is on-topic yet wrong, and the model has no signal that the context is stale or off-target.
When retrieval fails, RAG does not degrade gracefully — it fails upward, laundering a bad passage into a fluent, cited-looking answer. CRAG’s premise is that the fix is not a bigger retriever or a bigger generator but an explicit quality check on what came back, plus a defined recovery move for each quality level. It treats retrieval as a component that can be wrong and builds a correction step around that fact, rather than pretending the top-k is always usable.
The loop in one pass
CRAG is a plug-in wrapper around any existing retriever and generator. The control flow for a single query is:
docs = retrieve(query, k)
scores = [evaluator(query, d) for d in docs]
conf = aggregate(scores) # one number for the batch
if conf >= upper: action = CORRECT # refine internal docs
elif conf < lower: action = INCORRECT # discard, web search
else: action = AMBIGUOUS # do both, then merge
knowledge = build_context(action, docs, query)
answer = generate(query, knowledge)Everything interesting lives in two places: the evaluator that turns a (query, document) pair into a confidence score, and the three-way branch that decides what corrective knowledge to assemble. The generator itself is untouched — CRAG improves the context it receives, not the model. That decoupling is why it can bolt onto a pipeline you already have.
The retrieval evaluator
The heart of CRAG is a lightweight retrieval evaluator: a small model, in the original work a fine-tuned T5-large, that takes a query and one retrieved document and emits a scalar relevance confidence score. Crucially it is small — hundreds of millions of parameters, not billions — so scoring the k retrieved passages adds a modest, predictable cost rather than a second full LLM pass.
It is trained as a relevance discriminator: positive examples are genuinely relevant query-document pairs, negatives are hard, plausibly similar but off-target pairs. The output is squashed to a bounded range — think of it living in [-1, 1], where strongly positive means ‘this document answers the query’ and strongly negative means ‘this is a distractor.’ Because it is a dedicated judge rather than the generator grading its own homework, it gives an independent read on retrieval quality — the signal ordinary RAG throws away.
Two thresholds, three actions
A per-document score has to become one decision for the query. CRAG aggregates the individual scores — in practice the top document’s score dominates the verdict — and compares the result against two cut points, an upper threshold and a lower threshold, carving the confidence line into three zones:
| Verdict | Condition | Corrective action |
|---|---|---|
| Correct | conf ≥ upper | Keep docs, but refine them into clean knowledge strips |
| Ambiguous | lower ≤ conf < upper | Refine the docs and pull web knowledge, then merge |
| Incorrect | conf < lower | Discard all docs, replace with web-search results |
The two-threshold design is deliberate. A single cut would force a hard correct/incorrect flip right at the boundary, exactly where the evaluator is least certain. The middle ambiguous band is a hedge: near the boundary CRAG refuses to bet everything on either the internal corpus or the open web, and instead uses both so a wrong guess on one side is cushioned by the other.
A worked confidence example
Suppose k = 3 and the evaluator returns, for the query ‘What is the 2024 EU AI Act risk tiering?’, scores [0.71, 0.12, -0.34] for the three retrieved passages. Take the aggregate as the max, conf = 0.71. With thresholds upper = 0.59 and lower = -0.99, we have 0.71 ≥ 0.59, so the verdict is Correct.
But ‘Correct’ does not mean ‘paste all three passages.’ Only the first passage scored well; the other two are near-zero and negative noise. This is precisely why the correct branch still runs refinement — it must strip the two weak passages and even prune irrelevant sentences inside the strong one. Now shift the scores to [0.10, -0.20, -0.35]: the max is 0.10. If upper = 0.59 and lower = -0.10, then -0.10 ≤ 0.10 < 0.59 lands in the ambiguous band, and CRAG augments the thin internal signal with web results.
Knowledge refinement: decompose then recompose
Even a relevant document is mostly filler around a few load-bearing sentences. CRAG’s knowledge refinement extracts the signal with a decompose-then-recompose routine. First decompose: split each retrieved document into fine-grained segments the paper calls knowledge strips — roughly a sentence or a couple of sentences each. Then score every strip with the same evaluator, discard the strips that fall below the relevance bar, and recompose the survivors in order into a compact context.
The effect is a second, finer filter operating one level below the document. Retrieval decides which documents; refinement decides which sentences within them. That matters for a small generator with a tight context window: it spends its limited attention on dense, on-target knowledge instead of diluting it across paragraphs of boilerplate. Less irrelevant context also means fewer distractor tokens the model can anchor a hallucination to.
The web-search fallback
When the verdict is incorrect, the internal corpus simply does not contain the answer, and refining garbage yields refined garbage. CRAG’s recovery is to leave the corpus entirely and issue a web search. The original query — often phrased for a human, not a search engine — is first rewritten into keyword-style queries, submitted to a search API, and the returned pages are fetched and run through the same decompose-recompose refinement so the external text arrives in the same clean-strip form.
This is what makes CRAG corrective rather than merely evaluative: detecting bad retrieval is only half the job; the other half is having somewhere better to go. The web acts as an open-domain backstop for exactly the queries a fixed corpus was never going to cover — recent events, long-tail facts, anything outside the indexed set. In the ambiguous band the same web knowledge is merged with the refined internal strips rather than replacing them.
Why the correction actually helps
CRAG works because it breaks the failure chain at its weakest link. The core problem was that retrieval error propagated silently into generation; the evaluator makes that error observable, and the three actions give the pipeline somewhere to go once it is observed. Instead of one path — retrieve, then generate no matter what — there are recovery routes graded to how bad retrieval was.
It is also strictly modular. The evaluator is a separate small model, the refinement is a deterministic split-score-filter, and the web search is an external call — none of them requires retraining or even touching the generator. That plug-and-play property is a large part of the appeal: you can wrap CRAG around an existing retriever-plus-LLM stack and get graceful degradation on retrieval misses, which is the exact regime where naive RAG is most dangerous and most confidently wrong.
What the correction costs
The corrections are not free, and the bill has two very different line items. The evaluator pass is cheap and predictable: scoring k documents (and then the handful of knowledge strips) with a few-hundred-million-parameter model is a small fraction of one generator forward pass, and it runs on the same hardware. Refinement adds only a linear scan over strips.
The web-search fallback is the expensive branch. It leaves your infrastructure entirely — a network round trip to a search API, page fetches, and re-refinement — adding hundreds of milliseconds to seconds of latency and an external dependency that can rate-limit or fail. The good news is that this cost is conditional: it fires only when the evaluator judges retrieval too poor to use, so on a well-covered corpus most queries take the cheap correct/refine path and only the genuine misses pay the web toll.
CRAG on a CPU-bound small model
CRAG is unusually friendly to the CPU / small-language-model setting, and the reason is a division of labor. A small quantized generator is far more brittle to bad context than a frontier model — it has less internal knowledge to override a misleading passage, so a distractor is more likely to steer it wrong. Feeding it pre-filtered, refined context is therefore worth more, not less, than it is for a large model.
And the extra work is offloaded to a cheap component: the retrieval evaluator is a small discriminative model that runs comfortably on CPU, so you spend a little classifier compute to protect the scarce, expensive generator tokens. The tighter refined context also shortens the prompt, which on CPU — where prefill and per-token cost bite hard — directly reduces latency. You pay a small, fixed CPU tax to avoid feeding your weakest link its worst input.
Pitfalls and tuning
CRAG’s quality rides on the evaluator and its two thresholds, and both are easy to get wrong. The evaluator is a trained model with its own error rate: a domain far from its training distribution produces miscalibrated scores, and every downstream decision inherits that miscalibration. The thresholds are a genuine trade-off — set upper too high and you trigger needless web searches on perfectly good retrievals (slow, and you import web noise); set lower too low and you never escape a bad corpus.
Two more traps. The web fallback trusts the open internet, which can be lower-quality or contradictory — the refinement step is your only filter, so it matters. And max-based aggregation means one high-scoring document flips the whole query to correct, which is usually right but hides the case where that top document is a confident false positive. Calibrate the thresholds on held-out data for your corpus rather than trusting defaults.