SPLADE Math

The gap SPLADE is built to close

Exact-match retrieval scores a document by the query terms it literally contains. A query for leaky faucet and a document about a dripping tap share almost no surface tokens, so the lexical score is near zero no matter how relevant the document is. That is vocabulary mismatch, and it is the structural weakness of every term-frequency scorer.

Dense bi-encoders fix it by abandoning terms entirely: both sides become points in R^d and semantic proximity does the work. But you inherit a new set of problems — the representation is uninterpretable, exact rare-token matches are no longer guaranteed, and you need an approximate-nearest-neighbour index rather than the inverted index your search stack already runs. SPLADE takes the third path: stay in vocabulary space, keep the index, and let the model learn which terms belong in the representation and how much each is worth.

The MLM head as a projection into vocabulary space

The mechanism is a piece of machinery that already exists. Run the encoder over an input of N tokens and you get hidden states H: [N, d]. BERT’s pretraining head projects each of those back onto the vocabulary to predict masked tokens:

w_ij = transform(h_i)·E_j + b_j
H: [N, d]   E: [|V|, d]   W: [N, |V|]

E is the (tied) input embedding matrix and b the output bias, so W holds, for every position i, a logit for every vocabulary entry j. With BERT’s WordPiece vocabulary |V| = 30522, a 128-token passage yields a 128 × 30522 matrix. Crucially the logit at position i for token j is not a term-frequency count: it is a contextual judgement of how much token j belongs at that spot. That is what lets the word faucet light up over the word tap.

Log saturation, and why not raw weights

W is dense and unbounded, so it needs a transform that produces exact zeros and tames outliers. SPLADE uses:

φ(w) = log(1 + ReLU(w))

Two jobs, one function. ReLU hard-zeros every negative logit — the vast majority — which is where sparsity physically comes from; a soft transform such as a sigmoid would leave 30k tiny nonzeros and no index. log(1 + ·) is concave, so it saturates: a logit of 8.0 becomes 2.197 and a logit of 3.0 becomes 1.386, collapsing a 2.7× ratio to 1.59×. Without it, one confidently predicted term would dominate the dot product and rank documents on a single dimension. The intuition matches term-frequency saturation in BM25 — the tenth occurrence of a word means less than the second — except here it is applied to a learned relevance logit rather than a count, and it is differentiable almost everywhere so gradients still flow.

Max pooling over positions

φ(W) is still [N, |V|]; the representation must be one vector of length |V|. SPLADE pools over positions:

w_j = max_{i=1..N} log(1 + ReLU(w_ij))

Version one of the model summed instead; version two switched to max and measured a consistent gain. The reason is that sum conflates two different signals. A term weakly implied at forty positions accumulates the same mass as a term strongly implied once, so long documents inflate everything and the length normalisation problem BM25 solves with b comes back through the side door. Max asks a cleaner question: is there any point in this document where token j is strongly implied? The output is a vector of shape [|V|] that is naturally length-robust, and each nonzero traces back to the single position responsible for it — which is also why the representation is debuggable.

Term expansion, and why it stays readable

The payoff is expansion. Because the head scores every vocabulary entry rather than only the ones present, a passage reading “fix a dripping tap by replacing the washer” will carry a substantial weight on faucet, leak, plumbing and repair — none of which appear in it. Matching is no longer gated on surface overlap; it is gated on learned lexical implication, and the vocabulary-mismatch problem largely dissolves.

What separates this from a dense embedding is that the axes never stop meaning anything. Dimension j is vocabulary token j, so the representation inverts trivially: sort the nonzeros by weight and you can read the model’s expansion as a ranked term list. A bad ranking traces back to a specific overweighted term, expansions can be audited before deployment, and nothing about the vector needs a nearest-neighbour structure to search — it is still a bag of weighted words.

A worked scoring example

Scoring is a plain sparse dot product over the shared vocabulary:

s(q, d) = Σ_{j ∈ V} w_j^q · w_j^d

Take the query how to fix a leaky faucet and a passage about replacing a washer in a dripping tap. Showing each side’s top terms only:

The total is 10.19. The instructive part is that literal overlap between the two surface texts is the single word fix, worth 0.600 — 94% of the score comes from term pairs that expansion created on one side or the other. Note also that leaky contributes nothing: the document side expanded to the lemma leak, not the adjective.

The FLOPS regularizer

Nothing so far forces sparsity to be useful. Training adds a penalty. For a batch of B examples, let a_j be the mean weight of token j across the batch; the FLOPS surrogate is

a_j = (1/B) Σ_b w_j^(b)
L_FLOPS = Σ_j (a_j)^2
L = L_rank + λ_q·L_FLOPS^q + λ_d·L_FLOPS^d

The square is the whole trick, and it is why L1 alone is not enough. Consider 1000 documents over a four-token vocabulary, each spending a total weight of 1.0. If every document puts that mass on the same token, a = (1, 0, 0, 0) and L_FLOPS = 1.0; if they spread across the four, a = (.25, .25, .25, .25) and L_FLOPS = 0.25. Identical L1 mass, 4× difference in penalty — and in reality, one posting list of length 1000 versus four of length 250. FLOPS prices collision, which is what actually costs query time. Separate λ values let you make queries much sparser than documents, and both are usually ramped in quadratically over warmup so the model learns to rank before it is told to shut up.