Late interaction is the idea that you should not squash a query or a document into a single vector before you compare them. Standard dense retrieval encodes each text as one pooled embedding and scores a pair with one dot product — fast, but it forces every nuance of a passage through a single bottleneck. ColBERT keeps one embedding per token on both sides and defers their interaction to scoring time, where a simple operator called MaxSim lets each query term find its best match anywhere in the document. That single change — per-token vectors plus MaxSim — recovers much of the fine-grained term matching that pooling throws away, while staying far cheaper than running a full cross-encoder over every candidate. This piece works through the vectors, the MaxSim formula with a numeric example, the storage cost that per-token embeddings create, the PLAID and compression tricks that tame it, and exactly where late interaction sits between single-vector dense retrieval and cross-encoder reranking.
The pooling bottleneck in single-vector retrieval
Classic dense retrieval — DPR, most sentence-transformer bi-encoders — runs a query q and a document d through an encoder and pools the per-token hidden states into one vector each: E_q ∈ R^k and E_d ∈ R^k (mean pooling, or the [CLS] vector). Relevance is a single dot product, score = E_q · E_d. This is wonderfully cheap: documents are encoded once, offline, and search becomes approximate nearest-neighbor over one vector per document.
The cost is expressive. A 400-token passage might mention a rare entity, a date, and a technical term, each decisive for some query — but all of that must be averaged into k numbers (often 768). Pooling is a lossy summary: signals that matter to one query get diluted by everything else in the passage. The result is the well-known weakness of single-vector models on queries that hinge on a specific term the passage mentions only once. That is precisely the gap late interaction is built to close.
Per-token embeddings: keep the bag, defer the match
ColBERT drops the pooling step. It encodes the query into a set of token embeddings E_q = {q_1, …, q_n} and the document into E_d = {d_1, …, d_m}, each q_i, d_j ∈ R^k and each L2-normalized so that a dot product equals cosine similarity. Typical ColBERT uses a compact k = 128 per token — much smaller than the 768-dim single vector, deliberately, because there are now many of them.
Crucially, the query and document encoders never see each other during encoding: documents are still encoded offline and independently, exactly as in a bi-encoder. What changes is that the interaction between the two texts is delayed to scoring time — hence ‘late interaction.’ The model keeps the full bag of contextualized token vectors and only decides how they line up when an actual query arrives. This is the structural middle ground: bi-encoder-style precomputation, but with the per-token detail that a pooled vector destroys.
MaxSim: the scoring formula
The late-interaction operator is MaxSim. For each query token, take its cosine similarity to every document token, keep the maximum, and then sum those maxima over the query tokens:
S(q, d) = Σ_{i=1..n} max_{j=1..m} q_i · d_j
q_i, d_j are L2-normalized => q_i · d_j = cos(q_i, d_j) ∈ [-1, 1]
n = # query tokens, m = # document tokensRead it as: every query term independently finds its single best matching term in the document, and the document’s score is how well its best matches add up. The inner max is a soft, embedding-space version of ‘does this term appear here?’ — it fires on synonyms and morphological variants, not just exact strings, because the vectors are contextual. The outer Σ rewards documents that cover more of the query’s terms well. Note the asymmetry: we max over document tokens for each query token, so a long document is not penalized for containing extra tokens — irrelevant document tokens simply never win a max.
A worked example
Take a two-token query, q_1 = ‘jaguar’, q_2 = ‘speed’, and a three-token document with tokens d_1 = ‘cheetah’, d_2 = ‘fastest’, d_3 = ‘cat’. Suppose the normalized embeddings give this cosine matrix:
| cos(q_i, d_j) | d_1 cheetah | d_2 fastest | d_3 cat | row max |
|---|---|---|---|---|
| q_1 jaguar | 0.71 | 0.12 | 0.66 | 0.71 |
| q_2 speed | 0.20 | 0.83 | 0.05 | 0.83 |
MaxSim keeps each row’s maximum — 0.71 for jaguar (best matched by cheetah, a big cat) and 0.83 for speed (best matched by fastest) — and sums them: S(q, d) = 0.71 + 0.83 = 1.54. Each query term was independently satisfied by a different, semantically related document term. A pooled bi-encoder would instead average {cheetah, fastest, cat} into one vector and compare it to the average of {jaguar, speed}; the sharp 0.83 alignment on speed↔fastest gets smeared into a mushier number, and the document may lose to a passage that is vaguely on-topic everywhere but decisive nowhere.
Why late interaction recovers fine-grained matching
The win comes from where the max lives. In a single-vector model the only comparison happens after both texts are already collapsed, so a strong term alignment cannot express itself — it has been averaged away before scoring. MaxSim moves the comparison to the token grid, so a single excellent alignment (0.83) survives at full strength instead of being diluted by the many mediocre alignments around it.
This makes ColBERT behave a little like a soft, semantic version of a term-matching system such as BM25: it rewards documents that actually contain (the embedding of) each query term, term by term. But unlike BM25 it matches meaning — speed matches fastest, jaguar matches cheetah — because the vectors are contextual. Empirically this gives strong out-of-domain robustness: late interaction generalizes to new corpora better than single-vector models, which tend to overfit the pooled-similarity distribution they were trained on. You get lexical-style precision with embedding-style recall.
Not a cross-encoder: interaction without joint encoding
It is tempting to lump ColBERT with cross-encoders, since both ‘let query and document interact.’ The difference is decisive for cost. A cross-encoder concatenates query and document and runs them together through the full transformer, so every query token attends to every document token through all layers. That is the most accurate scorer we have — and the most expensive: nothing can be precomputed, because the document’s representation depends on the query. You must run a full forward pass per (query, document) pair, which limits it to re-ranking a few dozen candidates.
ColBERT’s interaction is only the MaxSim dot products at the very end. The expensive transformer encoding of the document happened offline, query-independent. So late interaction is far cheaper than a cross-encoder while capturing much of its term-level sensitivity — it interacts token embeddings, not whole networks. The spectrum runs: bi-encoder (interact once, pooled) < ColBERT (interact per-token, cheap) < cross-encoder (interact per-token, through every layer, costly).
The storage cost: many vectors per document
Late interaction has one blunt price: you store an embedding for every token, not one per document. A corpus of 10M passages averaging 130 tokens each holds ~1.3 billion token vectors. Even at a compact k = 128 dimensions in fp16, that is 1.3e9 × 128 × 2 bytes ≈ 330 GB — against roughly 10e6 × 128 × 2 ≈ 2.5 GB for a single-vector index over the same passages. Two orders of magnitude.
The blow-up is inherent to the method: fine-grained matching needs fine-grained representation, and that means keeping the tokens. It also reshapes search — you are no longer doing one nearest-neighbor lookup per document but reasoning over a huge pool of token vectors. Left naive, this bloats the index and slows retrieval, which is why practical ColBERT is really ColBERT plus an engineering stack — quantization and PLAID — whose entire job is to make the multi-vector index affordable to store and fast to search.
Taming it: quantization, pruning, and PLAID
Three levers shrink the cost. Dimension: ColBERT already uses a small k = 128 instead of 768. Quantization: ColBERTv2 introduces residual compression — each token vector is approximated by a nearby centroid plus a low-bit residual (1–2 bits per dimension), cutting storage several-fold with little quality loss, because many token embeddings cluster tightly. Pruning: some tokens (punctuation, stopwords) carry little signal and can be dropped from the index.
PLAID is the retrieval engine that makes the compressed index fast. It exploits the centroids from residual compression as a coarse first filter: instead of scanning every token vector, PLAID uses centroid identities to quickly shortlist candidate passages, aggressively prunes passages that cannot score well, and only then decompresses the survivors for exact MaxSim. The effect is that a billion-vector index becomes searchable in tens of milliseconds. Storage and latency, the two objections to late interaction, are largely answered by compression plus centroid-guided pruning — not by changing the MaxSim math.
Two-stage retrieval with MaxSim
ColBERT is unusual in that MaxSim serves in both retrieval stages, not only re-ranking. Stage 1 (candidate generation): every query token vector is used to probe the token index for its nearest document tokens (approximate nearest-neighbor over the token pool). The union of documents that own those matched tokens becomes the candidate set — a token-level, recall-oriented filter that already reflects the ‘each term finds its best match’ logic.
Stage 2 (scoring): each candidate document is scored with the full MaxSim sum S(q, d) = Σ_i max_j q_i · d_j over its token vectors, and the list is ranked by that score. Because both stages use the same operator, retrieval is coherent end-to-end — there is no separate, mismatched first-stage retriever whose notion of relevance disagrees with the scorer. This is the flip side of a cross-encoder, which can only ever re-rank a candidate list produced by some other retriever; ColBERT can retrieve from the whole corpus on its own, then score, with one consistent similarity.
Practical notes and pitfalls
A few things bite in practice. Normalization is load-bearing: MaxSim assumes vectors are L2-normalized so the dot product is a bounded cosine; skip it and the max is dominated by vector magnitude, not direction. Score is length-sensitive: because MaxSim sums over query tokens, raw scores are not comparable across queries of different length — fine for ranking within one query, but do not threshold across queries without normalizing by n. Query augmentation: ColBERT pads short queries with [MASK] tokens that act as learnable query expansion, contributing extra max terms — a detail worth knowing when you reason about the sum.
For CPU or small-model deployment, the trade is clear-eyed. Encoding is a normal transformer forward pass, but scoring is just dot products and maxima, so MaxSim itself is cheap even on CPU once the index is compressed. The real cost is memory and index engineering — exactly where ColBERTv2 and PLAID earn their place, turning an appealing operator into a system you can run over millions of documents.