The one rule: only biases move

Every linear layer in a transformer computes y = W x + b. Full fine-tuning updates both W and b. BitFit keeps W frozen at its pretrained value and lets gradient descent touch only b (together with the final classification or regression head, which is new and unavoidably trained). Introduced by Ben Zaken, Ravfogel, and Goldberg in 2022, the method was a stress test of a hypothesis: that pretraining already installs the useful directions of computation in the weight matrices, and adapting to a downstream task is mostly a matter of re-thresholding — shifting where each neuron sits relative to its nonlinearity — rather than learning new transformations.

Practically this makes the update set trivially easy to define: walk the model, collect every parameter whose name ends in .bias, set requires_grad=True on those and False on everything else. No new modules are inserted, no matrices are added, the architecture is byte-for-byte unchanged. That structural simplicity is BitFit’s signature and the source of both its charm and its ceiling.

Advertisement

Exactly which biases exist in a transformer block

To count and reason about BitFit you have to know where biases actually live. In a standard post-norm encoder block (BERT-style) each layer carries these additive vectors, all of length d unless noted:

  • Attention projections — b_Q, b_K, b_V on the query/key/value maps, plus b_O on the attention output projection.
  • Feed-forward — b_1 on the intermediate expansion (length d_ff, typically 4d) and b_2 on the contraction back to d.
  • LayerNorm — the shift term β (a genuine additive bias) on each of the two LayerNorms.

One subtlety worth stating cleanly: LayerNorm also has a gain γ, but γ is a multiplicative scale, not a bias. Strict BitFit trains only the additive terms, so γ and every weight matrix stay frozen. The query bias b_Q and the second feed-forward bias b_2 turned out, in the original ablations, to carry most of the adaptation on their own.

Advertisement

Counting the parameters on BERT-base

Numbers make the ‘one in a thousand’ claim concrete. BERT-base has d = 768, d_ff = 3072, and 12 layers. Summing the bias lengths in one block:

b_Q + b_K + b_V + b_O = 4 × 768        = 3072
b_1 (intermediate)          = 3072
b_2 (output)                =  768
LayerNorm β_1 + β_2        = 2 × 768        = 1536
---------------------------------------------
per layer                   = 8448 params
× 12 layers               ≈ 101,000 params

Add the embedding LayerNorm bias and a pooler bias and you are at roughly 0.1M trainable parameters against BERT-base’s 110M — about 0.09%, so under one in a thousand. Even that overstates the storage cost: to ship a task you save only the changed bias vectors, a file on the order of tens of kilobytes rather than the ~440 MB of a full checkpoint. Ten downstream tasks cost ten tiny bias deltas over one shared frozen backbone.

What a bias can actually do to a layer

The expressivity question is the whole game, and it has a clean answer. For a frozen linear map, the reachable function family under BitFit is exactly { x → W x + b : b ∈ R^d }. The Jacobian ∂y/∂x = W is fixed — BitFit cannot rotate, rescale, or reweight the input directions W extracts. All it controls is the constant offset. Geometrically, the pre-activation for every input is translated by the same vector b; the shape of the transformation is untouched and only its position moves.

That offset is not powerless, because it sits before a nonlinearity. A bias shifts each neuron’s pre-activation across its ReLU/GELU threshold, effectively retuning which inputs switch the unit on and how hard. So BitFit’s real lever is re-thresholding a fixed feature bank: it decides how eagerly each pretrained feature fires, but it cannot invent a feature the frozen weights do not already compute. Adaptation happens by re-gating, not by re-learning.

A worked example: a query bias in attention

Follow one bias through attention to see the effect concretely. A query is q = x W_Q + b_Q and an attention logit against key k is the dot product q · k (before the 1/√d_k scale). Shift the query bias by Δ and the logit becomes:

(q + Δ) · k = q · k + Δ · k

The added term Δ · k depends only on the key, not on the query position. So tuning b_Q installs a content-based prior: it pushes attention uniformly toward (or away from) keys that align with Δ, regardless of which token is doing the attending. That is a genuine, useful knob — a task can learn ‘attend a little more to punctuation-like keys everywhere’ — but notice what it cannot do: it cannot make the query-dependent pattern of attention different, because W_Q and W_K, which decide how token content maps to the matching geometry, are frozen. BitFit can bias the routing; it cannot rewire it.

Why it works better than it has any right to

It is genuinely surprising that adjusting 0.09% of parameters recovers most of full fine-tuning on GLUE. The leading explanation is that large-scale masked-language pretraining already learns broadly reusable features, and many classification tasks are, at the feature level, close to what pretraining saw. Adaptation then really is mostly a matter of re-weighting and re-thresholding existing features for the new label set — precisely the operation a bias can perform. The task-specific head does the final linear recombination; BitFit’s biases tune the gating underneath it.

There is a regularization angle too. With so few degrees of freedom, BitFit cannot overfit a small dataset by memorizing through the backbone, and it cannot drift far from the pretrained solution — the frozen weights anchor it. On small and medium data this constraint is a feature, and it partly explains why BitFit is often most competitive in exactly the low-resource regime where full fine-tuning is most prone to overfit.

The expressivity ceiling, stated honestly

The same frozen-Jacobian property that makes BitFit elegant also caps it. Because the reachable family only translates activations, BitFit is weak whenever a task needs genuinely new feature directions or new interactions the pretrained weights never learned. Large domain shifts — adapting a general-English encoder to protein sequences, code, or a very different language — ask for changes to W that no choice of b can supply. On harder generation and reasoning tasks, and as backbones grow, the gap to full fine-tuning and to LoRA tends to widen.

This is the crisp line between the PEFT methods. LoRA adds a low-rank ΔW = BA, so it can move the transformation itself within a rank-r subspace. Adapters insert a small nonlinear bottleneck, adding new learned computation. BitFit adds neither — it is strictly a shift of a fixed map. That makes it the least expressive of the three by construction, which is exactly why it is the cleanest baseline: it measures how much of a task is solvable by re-gating alone.