SPLADE Math

The gap SPLADE is built to close

Exact-match retrieval scores a document by the query terms it literally contains. A query for leaky faucet and a document about a dripping tap share almost no surface tokens, so the lexical score is near zero no matter how relevant the document is. That is vocabulary mismatch, and it is the structural weakness of every term-frequency scorer.

Dense bi-encoders fix it by abandoning terms entirely: both sides become points in R^d and semantic proximity does the work. But you inherit a new set of problems — the representation is uninterpretable, exact rare-token matches are no longer guaranteed, and you need an approximate-nearest-neighbour index rather than the inverted index your search stack already runs. SPLADE takes the third path: stay in vocabulary space, keep the index, and let the model learn which terms belong in the representation and how much each is worth.

The MLM head as a projection into vocabulary space

The mechanism is a piece of machinery that already exists. Run the encoder over an input of N tokens and you get hidden states H: [N, d]. BERT’s pretraining head projects each of those back onto the vocabulary to predict masked tokens:

w_ij = transform(h_i)·E_j + b_j
H: [N, d]   E: [|V|, d]   W: [N, |V|]

E is the (tied) input embedding matrix and b the output bias, so W holds, for every position i, a logit for every vocabulary entry j. With BERT’s WordPiece vocabulary |V| = 30522, a 128-token passage yields a 128 × 30522 matrix. Crucially the logit at position i for token j is not a term-frequency count: it is a contextual judgement of how much token j belongs at that spot. That is what lets the word faucet light up over the word tap.

Log saturation, and why not raw weights

W is dense and unbounded, so it needs a transform that produces exact zeros and tames outliers. SPLADE uses:

φ(w) = log(1 + ReLU(w))

Two jobs, one function. ReLU hard-zeros every negative logit — the vast majority — which is where sparsity physically comes from; a soft transform such as a sigmoid would leave 30k tiny nonzeros and no index. log(1 + ·) is concave, so it saturates: a logit of 8.0 becomes 2.197 and a logit of 3.0 becomes 1.386, collapsing a 2.7× ratio to 1.59×. Without it, one confidently predicted term would dominate the dot product and rank documents on a single dimension. The intuition matches term-frequency saturation in BM25 — the tenth occurrence of a word means less than the second — except here it is applied to a learned relevance logit rather than a count, and it is differentiable almost everywhere so gradients still flow.

Max pooling over positions

φ(W) is still [N, |V|]; the representation must be one vector of length |V|. SPLADE pools over positions:

w_j = max_{i=1..N} log(1 + ReLU(w_ij))

Version one of the model summed instead; version two switched to max and measured a consistent gain. The reason is that sum conflates two different signals. A term weakly implied at forty positions accumulates the same mass as a term strongly implied once, so long documents inflate everything and the length normalisation problem BM25 solves with b comes back through the side door. Max asks a cleaner question: is there any point in this document where token j is strongly implied? The output is a vector of shape [|V|] that is naturally length-robust, and each nonzero traces back to the single position responsible for it — which is also why the representation is debuggable.

Term expansion, and why it stays readable

The payoff is expansion. Because the head scores every vocabulary entry rather than only the ones present, a passage reading “fix a dripping tap by replacing the washer” will carry a substantial weight on faucet, leak, plumbing and repair — none of which appear in it. Matching is no longer gated on surface overlap; it is gated on learned lexical implication, and the vocabulary-mismatch problem largely dissolves.

What separates this from a dense embedding is that the axes never stop meaning anything. Dimension j is vocabulary token j, so the representation inverts trivially: sort the nonzeros by weight and you can read the model’s expansion as a ranked term list. A bad ranking traces back to a specific overweighted term, expansions can be audited before deployment, and nothing about the vector needs a nearest-neighbour structure to search — it is still a bag of weighted words.

A worked scoring example

Scoring is a plain sparse dot product over the shared vocabulary:

s(q, d) = Σ_{j ∈ V} w_j^q · w_j^d

Take the query how to fix a leaky faucet and a passage about replacing a washer in a dripping tap. Showing each side’s top terms only:

The total is 10.19. The instructive part is that literal overlap between the two surface texts is the single word fix, worth 0.600 — 94% of the score comes from term pairs that expansion created on one side or the other. Note also that leaky contributes nothing: the document side expanded to the lemma leak, not the adjective.

The FLOPS regularizer

Nothing so far forces sparsity to be useful. Training adds a penalty. For a batch of B examples, let a_j be the mean weight of token j across the batch; the FLOPS surrogate is

a_j = (1/B) Σ_b w_j^(b)
L_FLOPS = Σ_j (a_j)^2
L = L_rank + λ_q·L_FLOPS^q + λ_d·L_FLOPS^d

The square is the whole trick, and it is why L1 alone is not enough. Consider 1000 documents over a four-token vocabulary, each spending a total weight of 1.0. If every document puts that mass on the same token, a = (1, 0, 0, 0) and L_FLOPS = 1.0; if they spread across the four, a = (.25, .25, .25, .25) and L_FLOPS = 0.25. Identical L1 mass, 4× difference in penalty — and in reality, one posting list of length 1000 versus four of length 250. FLOPS prices collision, which is what actually costs query time. Separate λ values let you make queries much sparser than documents, and both are usually ramped in quadratically over warmup so the model learns to rank before it is told to shut up.

Inverted-index mechanics, and where they strain

A max-pooled SPLADE vector is a term-weight list, so it drops into an existing inverted index almost unchanged: quantize each weight to an int8 impact, post it under its term, and score by traversing the query’s posting lists. A typical trained model leaves roughly 150–250 nonzeros per passage. At 8.8M passages and, say, five bytes per posting for a delta-coded docid plus one impact byte, ~1.7B postings land near 8 GB — comparable to an int8 dense index, and servable with ordinary CPU integer arithmetic.

The strain is in pruning. WAND, MaxScore and block-max skipping assume the sharply skewed weight distribution that IDF produces: a handful of rare, high-upper-bound terms let the traversal discard most candidates early. Learned weights are far flatter, so per-term upper bounds are loose and much less of each posting list can be skipped. Add an expanded query with 30–50 terms instead of five and the traversal work multiplies rather than prunes.

The efficiency-effectiveness curve

λ_q and λ_d are not settings you get right once — they sweep out a curve. Push them up and average nonzeros fall, posting lists shorten, latency drops and ranking quality decays; pull them down and quality climbs until the index is dense enough that you have lost the reason to be sparse. The honest way to report a model is a point on that curve: average nonzeros per query and per document, measured latency, and a ranking metric together. A quality number quoted without its sparsity number is meaningless, because it can always be bought.

The asymmetry is the useful insight. Query-side and document-side cost are not symmetric — document encoding is offline and amortized, query encoding and traversal are on the critical path. So the efficient variants of the model push λ_q hard, sometimes to the point of dropping query expansion entirely and letting the raw query terms meet a richly expanded index. Most of the recall benefit survives, and the online work collapses.

CPU realities and the pitfalls

On a CPU-only stack, traversal is rarely the bottleneck — the query encoder forward pass is. A BERT-base encoder is ~110M parameters, and running it on a short query costs tens of milliseconds on commodity cores before the index is even touched. That is the real argument for a distilled or tiny query encoder, or for the no-query-expansion configuration above, where the online cost falls to tokenization.

Four traps recur. Expansion is bounded by the tokenizer: a term absent from the WordPiece vocabulary can never be expanded onto, which bites hardest in specialized and multilingual corpora. Impact quantization must use the same scale at index and query time or scores silently distort. λ tuned on one corpus does not transfer — sparsity is a property of the trained distribution, not the architecture. And SPLADE scores are on no shared scale with BM25 or cosine, so combining rankings needs rank-based fusion, not addition.