Retrieval Architectures Overview

Retrieval as a factorized score

The most general way to score a query-document pair is a joint function s(q, d) that reads both texts together — the most accurate option and the least scalable, at one model forward pass per candidate.

Web-scale retrieval becomes possible by factorizing that score into two independent encoders:

s(q, d) ≈ f(q) · g(d)
f: query  → vector in R^d
g: document → vector in R^d

Because g(d) no longer depends on the query, every document vector is computed once, offline, and stored. At query time you encode once to get q: [1, d] and score the corpus D: [N, d] with one matrix-vector product D q^T → [N, 1]. Ranking has become linear algebra — with the weakness that comes attached: the query never meets the document’s words, only a fixed summary of them.

What the embedding space actually looks like

Embeddings are not spread evenly over the unit sphere in R^768. Transformer sentence embeddings are strongly anisotropic: they occupy a narrow cone, so two random texts typically have a cosine similarity of 0.6 to 0.8 rather than the near-zero you would expect in high dimensions.

Two consequences. First, absolute similarity scores are meaningless: a cosine of 0.72 is not ‘72% relevant’ and may sit below the corpus average. Only the ranking, and the gap between the top hit and the rest, carry information, so any hard-coded threshold breaks the moment you swap models. Second, individual coordinates mean nothing alone; the information lives in directions, which is why the similarity function you pick, and whether you normalize first, is not a detail.

Dot product, cosine and L2 — and when they coincide

Three similarity functions dominate vector search, for vectors a, b ∈ R^d:

dot(a, b)  = a · b        = Σ_i a_i b_i
cos(a, b)  = (a · b) / (||a|| ||b||)
L2(a, b)   = ||a − b||   = sqrt( Σ_i (a_i − b_i)^2 )

Expanding the squared Euclidean distance ties them together:

||a − b||^2 = ||a||^2 + ||b||^2 − 2 (a · b)

Now the precondition: if both vectors are L2-normalized so ||a|| = ||b|| = 1, then cos(a, b) = a · b and the identity collapses to ||a − b||^2 = 2 − 2 (a · b). Squared distance is a strictly decreasing affine function of the dot product, so ascending L2 gives exactly the order of descending dot or cosine. All three are interchangeable — but only after normalization. Without it, dot product rewards long vectors, which in a text corpus usually just means long or frequent documents.

A worked geometry example

A three-dimensional toy corpus. Query q = [1, 1, 0], ||q|| = √2 ≈ 1.414, two candidates:

d1 = [3.0, 0.0, 0.0]   ||d1|| = 3.000
d2 = [0.9, 0.9, 0.0]   ||d2|| = 1.273

dot(q, d1) = 3.0    cos(q, d1) = 3.0 / (1.414 × 3.000) = 0.707
dot(q, d2) = 1.8    cos(q, d2) = 1.8 / (1.414 × 1.273) = 1.000

The metrics disagree completely. Dot product puts d1 first because it is long; cosine puts d2 first because it points exactly along the query. Euclidean distance sides with cosine: ||q − d1||^2 = 5.0 versus ||q − d2||^2 = 0.02. Normalize at index build time and the disagreement vanishes — the identity predicts 2 − 2(0.707) = 0.586 and 2 − 2(1.0) = 0, the same order cosine gave.

Exact search: the flat-scan cost model

The baseline index is flat: all N vectors in an [N, d] array, every one scored. Two costs, only one of which matters:

FLOPs per query = 2 · N · d
Bytes per query = N · d · bytes_per_element
Arithmetic intensity = 2 FLOP / 4 B = 0.5 FLOP per byte  (fp32)

A CPU core issues roughly 5–20 floating-point operations per byte pulled from DRAM; a flat scan offers 0.5, so it is memory-bandwidth bound, not compute bound. For N = 1,000,000 chunks at d = 768 in fp32:

Index size = 1e6 × 768 × 4 B = 3.07 GB
FLOPs      = 2 × 1e6 × 768   = 1.54 GFLOP

At 20 GB/s, 50 GFLOP/s effective:
  memory  = 3.07 / 20 ≈ 154 ms per query
  compute = 1.54 / 50 ≈  31 ms per query

Memory time is five times compute time. Those constants are assumptions, but the ratio holds across commodity CPUs, so threads help only until the memory controller saturates and the real lever is moving fewer bytes. fp16 halves the index to 1.54 GB and latency to ~77 ms, int8 quarters it to ~38 ms, and halving d to 384 halves it again — a flat exact index stays viable to roughly 10^5 vectors on CPU.

ANN indexes and the recall-latency knob

Approximate nearest-neighbour indexes buy speed by giving up the guarantee of the true top-k. Three families, often combined:

With N = 10^6 and nlist = 4096, IVF at nprobe = 16 compares against 4,096 centroids plus about 3,900 cell members — roughly 8,000 distance computations instead of 1,000,000, a 125× reduction. Product quantization with m = 96 subvectors of 8 dimensions and 256 centroids each stores 96 bytes per vector rather than 3,072, shrinking that 3.07 GB index to 96 MB.

Every knob is the same dial renamed: how much of the index will you look at? Raising nprobe or efSearch lifts recall and latency together along a sharply concave curve — early increments are cheap, the last few percent are not. Tune it empirically: build a flat exact index over a sample, take its top-k as ground truth, sweep the knob, and pick the elbow.

Chunking sets the recall ceiling

The most consequential retrieval decision is made before any vector exists: how to cut documents into units. Each chunk becomes one point in R^d, produced by pooling its token representations. Pooling is averaging, and averaging dilutes.

Large chunks (1,000+ tokens) cover more ground per hit but produce muddy vectors: a passage spanning four topics lands near the centroid of all four and close to none, so a query matching one of them competes with the dilution of the rest. Small chunks (100–200 tokens) give sharp, well-separated vectors, but an answer spanning a boundary is split across two points, forcing a larger k to reassemble it.

Chunk size drives cost too: halving it doubles N, and so doubles index bytes and scan time. Two defaults survive contact with reality — cut on structural boundaries rather than a fixed token count, and overlap adjacent chunks by 10–20% so a sentence straddling a cut still appears whole somewhere.

Measuring retrieval: recall@k, MRR, nDCG

Three metrics answer three questions. Let R_q be the relevant set for query q.

recall@k = |R_q ∩ top_k| / |R_q|
MRR      = (1/|Q|) Σ_q  1 / rank of the FIRST relevant hit
DCG@k    = Σ_{i=1..k}  rel_i / log2(i + 1)
nDCG@k   = DCG@k / IDCG@k

recall@k asks whether the evidence reached the window at all; with one gold chunk per query it collapses to hit-rate@k. It matters most upstream: a chunk that is not retrieved cannot be recovered downstream. MRR averages 1/rank of the first hit, caring only about that one position — it is not MAP.

nDCG handles graded relevance with a logarithmic position discount; note the convention log2(i + 1) with i starting at 1, so rank 1 divides by log2(2) = 1. IDCG is the DCG of the perfectly ordered list, which pins nDCG to [0, 1].

A worked evaluation

Five queries, one gold chunk each, k = 5; the gold chunk lands at ranks 1, 3, not-in-top-5, 2 and 1.

recall@5 = 4 / 5 = 0.800
recall@1 = 2 / 5 = 0.400
MRR      = (1/1 + 1/3 + 0 + 1/2 + 1/1) / 5 = 2.833 / 5 = 0.567

Now graded relevance, one query whose top 5 score rel = [3, 0, 2, 3, 1]:

DCG@5  = 3/log2(2) + 0/log2(3) + 2/log2(4) + 3/log2(5) + 1/log2(6)
       = 3.000 + 0.000 + 1.000 + 1.292 + 0.387 = 5.679

ideal order = [3, 3, 2, 1, 0]
IDCG@5 = 3.000 + 1.893 + 1.000 + 0.431 + 0.000 = 6.323

nDCG@5 = 5.679 / 6.323 = 0.898

Read them together: recall@5 of 0.80 says one query in five is unanswerable whatever the generator does; MRR of 0.567 puts the gold chunk near but not at the top; nDCG of 0.898 says the ordering is good given what was retrieved — a healthy ranker on a leaky recall stage.

Pitfalls that quietly destroy recall

Asymmetric encoding. Many embedding models are trained with distinct query and passage roles and expect an instruction prefix; embedding a short question with the passage template puts it in the wrong region of the cone and costs recall with no error message. Metric mismatch. Indexing with inner product when the model was trained for cosine, without normalizing, silently reranks by document length.

Stale or mixed vectors. Embeddings from two model versions in one index are not comparable; changing models means a full re-index. Evaluating on the wrong axis. ANN recall measured against your own exact index reports fidelity to brute force, not whether brute force found the right documents — check against human labels too.

For a CPU-hosted small model the priority order is clear: normalize, keep d and N small, and quantize before you parallelize.