BitFit Theory

The one rule: only biases move

Every linear layer in a transformer computes y = W x + b. Full fine-tuning updates both W and b. BitFit keeps W frozen at its pretrained value and lets gradient descent touch only b (together with the final classification or regression head, which is new and unavoidably trained). Introduced by Ben Zaken, Ravfogel, and Goldberg in 2022, the method was a stress test of a hypothesis: that pretraining already installs the useful directions of computation in the weight matrices, and adapting to a downstream task is mostly a matter of re-thresholding — shifting where each neuron sits relative to its nonlinearity — rather than learning new transformations.

Practically this makes the update set trivially easy to define: walk the model, collect every parameter whose name ends in .bias, set requires_grad=True on those and False on everything else. No new modules are inserted, no matrices are added, the architecture is byte-for-byte unchanged. That structural simplicity is BitFit’s signature and the source of both its charm and its ceiling.

Exactly which biases exist in a transformer block

To count and reason about BitFit you have to know where biases actually live. In a standard post-norm encoder block (BERT-style) each layer carries these additive vectors, all of length d unless noted:

• Attention projections — b_Q, b_K, b_V on the query/key/value maps, plus b_O on the attention output projection.
• Feed-forward — b_1 on the intermediate expansion (length d_ff, typically 4d) and b_2 on the contraction back to d.
• LayerNorm — the shift term β (a genuine additive bias) on each of the two LayerNorms.

One subtlety worth stating cleanly: LayerNorm also has a gain γ, but γ is a multiplicative scale, not a bias. Strict BitFit trains only the additive terms, so γ and every weight matrix stay frozen. The query bias b_Q and the second feed-forward bias b_2 turned out, in the original ablations, to carry most of the adaptation on their own.

Counting the parameters on BERT-base

Numbers make the ‘one in a thousand’ claim concrete. BERT-base has d = 768, d_ff = 3072, and 12 layers. Summing the bias lengths in one block:

b_Q + b_K + b_V + b_O = 4 × 768        = 3072
b_1 (intermediate)          = 3072
b_2 (output)                =  768
LayerNorm β_1 + β_2        = 2 × 768        = 1536
---------------------------------------------
per layer                   = 8448 params
× 12 layers               ≈ 101,000 params

Add the embedding LayerNorm bias and a pooler bias and you are at roughly 0.1M trainable parameters against BERT-base’s 110M — about 0.09%, so under one in a thousand. Even that overstates the storage cost: to ship a task you save only the changed bias vectors, a file on the order of tens of kilobytes rather than the ~440 MB of a full checkpoint. Ten downstream tasks cost ten tiny bias deltas over one shared frozen backbone.

What a bias can actually do to a layer

The expressivity question is the whole game, and it has a clean answer. For a frozen linear map, the reachable function family under BitFit is exactly { x → W x + b : b ∈ R^d }. The Jacobian ∂y/∂x = W is fixed — BitFit cannot rotate, rescale, or reweight the input directions W extracts. All it controls is the constant offset. Geometrically, the pre-activation for every input is translated by the same vector b; the shape of the transformation is untouched and only its position moves.

That offset is not powerless, because it sits before a nonlinearity. A bias shifts each neuron’s pre-activation across its ReLU/GELU threshold, effectively retuning which inputs switch the unit on and how hard. So BitFit’s real lever is re-thresholding a fixed feature bank: it decides how eagerly each pretrained feature fires, but it cannot invent a feature the frozen weights do not already compute. Adaptation happens by re-gating, not by re-learning.

A worked example: a query bias in attention

Follow one bias through attention to see the effect concretely. A query is q = x W_Q + b_Q and an attention logit against key k is the dot product q · k (before the 1/√d_k scale). Shift the query bias by Δ and the logit becomes:

(q + Δ) · k = q · k + Δ · k

The added term Δ · k depends only on the key, not on the query position. So tuning b_Q installs a content-based prior: it pushes attention uniformly toward (or away from) keys that align with Δ, regardless of which token is doing the attending. That is a genuine, useful knob — a task can learn ‘attend a little more to punctuation-like keys everywhere’ — but notice what it cannot do: it cannot make the query-dependent pattern of attention different, because W_Q and W_K, which decide how token content maps to the matching geometry, are frozen. BitFit can bias the routing; it cannot rewire it.

Why it works better than it has any right to

It is genuinely surprising that adjusting 0.09% of parameters recovers most of full fine-tuning on GLUE. The leading explanation is that large-scale masked-language pretraining already learns broadly reusable features, and many classification tasks are, at the feature level, close to what pretraining saw. Adaptation then really is mostly a matter of re-weighting and re-thresholding existing features for the new label set — precisely the operation a bias can perform. The task-specific head does the final linear recombination; BitFit’s biases tune the gating underneath it.

There is a regularization angle too. With so few degrees of freedom, BitFit cannot overfit a small dataset by memorizing through the backbone, and it cannot drift far from the pretrained solution — the frozen weights anchor it. On small and medium data this constraint is a feature, and it partly explains why BitFit is often most competitive in exactly the low-resource regime where full fine-tuning is most prone to overfit.

The expressivity ceiling, stated honestly

The same frozen-Jacobian property that makes BitFit elegant also caps it. Because the reachable family only translates activations, BitFit is weak whenever a task needs genuinely new feature directions or new interactions the pretrained weights never learned. Large domain shifts — adapting a general-English encoder to protein sequences, code, or a very different language — ask for changes to W that no choice of b can supply. On harder generation and reasoning tasks, and as backbones grow, the gap to full fine-tuning and to LoRA tends to widen.

This is the crisp line between the PEFT methods. LoRA adds a low-rank ΔW = BA, so it can move the transformation itself within a rank-r subspace. Adapters insert a small nonlinear bottleneck, adding new learned computation. BitFit adds neither — it is strictly a shift of a fixed map. That makes it the least expressive of the three by construction, which is exactly why it is the cleanest baseline: it measures how much of a task is solvable by re-gating alone.

Cost profile: training, memory, inference

BitFit’s savings are real but it is worth being precise about which cost it cuts. It shrinks the optimizer and gradient footprint dramatically: with only ~0.1M trainable parameters, the Adam moment buffers and stored gradients cover those parameters alone, not all 110M, which is a large slice of fine-tuning’s memory. Checkpoints per task drop to kilobytes. Those are the wins.

What BitFit does not reduce is the forward and backward pass through the frozen backbone. You still run the full network and still backpropagate through every layer to reach the biases — the activations must be held for the backward pass just as in full fine-tuning. So BitFit is not primarily a compute-saver; it is a trainable-parameter and storage saver. At inference there is no overhead at all: a trained bias is folded back into the same W x + b the model already computes, so unlike an adapter there is no extra layer and no added latency — a real plus for a CPU-class deployment.

BitFit for a CPU-class small language model

On a small model destined for CPU inference, BitFit’s trade-offs read a little differently than on a big GPU-served encoder. The zero inference overhead is attractive — every millisecond matters on CPU, and BitFit adds none. Multi-task hosting is cheap: keep one frozen quantized backbone in memory and swap a few-kilobyte bias set per task, which is friendlier to a constrained device than loading separate full checkpoints. And the tiny trainable set means you can fine-tune on modest hardware without holding optimizer state for the whole model.

The caution is expressivity headroom. A small model has a thinner, less redundant feature bank than a large one, so ‘the features already exist, just re-gate them’ holds less reliably. If BitFit underperforms your target on a small SLM, that is the expected failure, not a bug — it is the signal to graduate to LoRA, which buys back the ability to adjust the transformation for a still-small parameter budget.

Common pitfalls and practical notes

A few things trip people up. First, the head: the new classification or regression head is trained in full and is not counted among the biases — forgetting to include it, or accidentally freezing it, breaks training outright. Second, learning rate: because there are so few parameters and the backbone is frozen, BitFit usually tolerates and benefits from a noticeably higher learning rate than full fine-tuning; the defaults tuned for updating W are often too timid.

Third, do not silently widen the definition. Training the LayerNorm gain γ or splicing in a low-rank term can help, but it is no longer BitFit — it is a hybrid, and reporting it as BitFit muddies the comparison the method exists to provide. Finally, treat BitFit as a diagnostic baseline: run it first, cheaply, and read the result. If it nearly matches full fine-tuning, your task is a re-gating task and you are done. If it lags badly, you have learned — equally cheaply — that the task needs new transformations, and LoRA is the next step.