The pooling bottleneck in single-vector retrieval
Classic dense retrieval — DPR, most sentence-transformer bi-encoders — runs a query q and a document d through an encoder and pools the per-token hidden states into one vector each: E_q ∈ R^k and E_d ∈ R^k (mean pooling, or the [CLS] vector). Relevance is a single dot product, score = E_q · E_d. This is wonderfully cheap: documents are encoded once, offline, and search becomes approximate nearest-neighbor over one vector per document.
The cost is expressive. A 400-token passage might mention a rare entity, a date, and a technical term, each decisive for some query — but all of that must be averaged into k numbers (often 768). Pooling is a lossy summary: signals that matter to one query get diluted by everything else in the passage. The result is the well-known weakness of single-vector models on queries that hinge on a specific term the passage mentions only once. That is precisely the gap late interaction is built to close.
Per-token embeddings: keep the bag, defer the match
ColBERT drops the pooling step. It encodes the query into a set of token embeddings E_q = {q_1, …, q_n} and the document into E_d = {d_1, …, d_m}, each q_i, d_j ∈ R^k and each L2-normalized so that a dot product equals cosine similarity. Typical ColBERT uses a compact k = 128 per token — much smaller than the 768-dim single vector, deliberately, because there are now many of them.
Crucially, the query and document encoders never see each other during encoding: documents are still encoded offline and independently, exactly as in a bi-encoder. What changes is that the interaction between the two texts is delayed to scoring time — hence ‘late interaction.’ The model keeps the full bag of contextualized token vectors and only decides how they line up when an actual query arrives. This is the structural middle ground: bi-encoder-style precomputation, but with the per-token detail that a pooled vector destroys.
MaxSim: the scoring formula
The late-interaction operator is MaxSim. For each query token, take its cosine similarity to every document token, keep the maximum, and then sum those maxima over the query tokens:
S(q, d) = Σ_{i=1..n} max_{j=1..m} q_i · d_j
q_i, d_j are L2-normalized => q_i · d_j = cos(q_i, d_j) ∈ [-1, 1]
n = # query tokens, m = # document tokensRead it as: every query term independently finds its single best matching term in the document, and the document’s score is how well its best matches add up. The inner max is a soft, embedding-space version of ‘does this term appear here?’ — it fires on synonyms and morphological variants, not just exact strings, because the vectors are contextual. The outer Σ rewards documents that cover more of the query’s terms well. Note the asymmetry: we max over document tokens for each query token, so a long document is not penalized for containing extra tokens — irrelevant document tokens simply never win a max.
A worked example
Take a two-token query, q_1 = ‘jaguar’, q_2 = ‘speed’, and a three-token document with tokens d_1 = ‘cheetah’, d_2 = ‘fastest’, d_3 = ‘cat’. Suppose the normalized embeddings give this cosine matrix:
| cos(q_i, d_j) | d_1 cheetah | d_2 fastest | d_3 cat | row max |
|---|---|---|---|---|
| q_1 jaguar | 0.71 | 0.12 | 0.66 | 0.71 |
| q_2 speed | 0.20 | 0.83 | 0.05 | 0.83 |
MaxSim keeps each row’s maximum — 0.71 for jaguar (best matched by cheetah, a big cat) and 0.83 for speed (best matched by fastest) — and sums them: S(q, d) = 0.71 + 0.83 = 1.54. Each query term was independently satisfied by a different, semantically related document term. A pooled bi-encoder would instead average {cheetah, fastest, cat} into one vector and compare it to the average of {jaguar, speed}; the sharp 0.83 alignment on speed↔fastest gets smeared into a mushier number, and the document may lose to a passage that is vaguely on-topic everywhere but decisive nowhere.
Why late interaction recovers fine-grained matching
The win comes from where the max lives. In a single-vector model the only comparison happens after both texts are already collapsed, so a strong term alignment cannot express itself — it has been averaged away before scoring. MaxSim moves the comparison to the token grid, so a single excellent alignment (0.83) survives at full strength instead of being diluted by the many mediocre alignments around it.
This makes ColBERT behave a little like a soft, semantic version of a term-matching system such as BM25: it rewards documents that actually contain (the embedding of) each query term, term by term. But unlike BM25 it matches meaning — speed matches fastest, jaguar matches cheetah — because the vectors are contextual. Empirically this gives strong out-of-domain robustness: late interaction generalizes to new corpora better than single-vector models, which tend to overfit the pooled-similarity distribution they were trained on. You get lexical-style precision with embedding-style recall.
Not a cross-encoder: interaction without joint encoding
It is tempting to lump ColBERT with cross-encoders, since both ‘let query and document interact.’ The difference is decisive for cost. A cross-encoder concatenates query and document and runs them together through the full transformer, so every query token attends to every document token through all layers. That is the most accurate scorer we have — and the most expensive: nothing can be precomputed, because the document’s representation depends on the query. You must run a full forward pass per (query, document) pair, which limits it to re-ranking a few dozen candidates.
ColBERT’s interaction is only the MaxSim dot products at the very end. The expensive transformer encoding of the document happened offline, query-independent. So late interaction is far cheaper than a cross-encoder while capturing much of its term-level sensitivity — it interacts token embeddings, not whole networks. The spectrum runs: bi-encoder (interact once, pooled) < ColBERT (interact per-token, cheap) < cross-encoder (interact per-token, through every layer, costly).
The storage cost: many vectors per document
Late interaction has one blunt price: you store an embedding for every token, not one per document. A corpus of 10M passages averaging 130 tokens each holds ~1.3 billion token vectors. Even at a compact k = 128 dimensions in fp16, that is 1.3e9 × 128 × 2 bytes ≈ 330 GB — against roughly 10e6 × 128 × 2 ≈ 2.5 GB for a single-vector index over the same passages. Two orders of magnitude.
The blow-up is inherent to the method: fine-grained matching needs fine-grained representation, and that means keeping the tokens. It also reshapes search — you are no longer doing one nearest-neighbor lookup per document but reasoning over a huge pool of token vectors. Left naive, this bloats the index and slows retrieval, which is why practical ColBERT is really ColBERT plus an engineering stack — quantization and PLAID — whose entire job is to make the multi-vector index affordable to store and fast to search.