Adapter Theory

Why adapt with a bottleneck at all

Full fine-tuning updates every parameter of a pretrained model. For a model with P parameters and T downstream tasks, you store T × P weights — 100 tasks on a 1B-parameter model means 100 billion stored weights, one full copy per task. The optimizer makes it worse during training: Adam keeps two moment buffers per weight, so fine-tuning touches roughly 3P values in memory before activations.

The adapter bet is that task adaptation is a low-dimensional correction to a good general-purpose representation. Instead of moving all of P, freeze the backbone and insert a small trainable module in each layer that nudges hidden states toward the task. If the module has p « P parameters, each new task costs only p extra weights, gradients flow only into p values, and one frozen backbone serves every task. Houlsby et al. showed this matches full fine-tuning on GLUE within ~0.4 points while training about 3% of the parameters — the empirical result that launched PEFT as a field.

The bottleneck: down-project, nonlinearity, up-project

An adapter is a two-layer MLP with a residual connection. Given a hidden state h ∈ R^d (per token), the adapter computes:

Adapter(h) = h + W_up · f(W_down · h + b_down) + b_up

W_down : [r, d]   down-projection,  d → r
f      : nonlinearity (GELU, ReLU, or SiLU)
W_up   : [d, r]   up-projection,    r → d
r « d        the bottleneck dimension

The down-projection compresses the d-dimensional state into an r-dimensional bottleneck; the nonlinearity lets the module represent task-specific corrections that are not linear in h; the up-projection maps the correction back to R^d so it can be added to the residual stream. The bottleneck is what makes the module cheap: both matrices are thin, [r, d] and [d, r], so parameters scale as 2rd rather than d². With r = d/16, the adapter is 32× smaller than one full d × d layer.

The nonlinearity is not optional

Drop the nonlinearity and the adapter collapses. Without f, the module computes h + W_up W_down h, and W_up W_down is a single rank-r linear map — which is precisely a LoRA update applied at that position. The nonlinearity is therefore the defining difference between a bottleneck adapter and LoRA: an adapter is a genuine little neural network computing a nonlinear function of the current hidden state, while LoRA is a static low-rank offset folded into a weight matrix.

This buys expressivity — a nonlinear correction can gate itself on the input, e.g. push formal-register tokens one way and code tokens another — but it is exactly what makes the adapter unmergeable. You cannot rewrite h + W_up f(W_down h) as W′ h for any fixed matrix W′, because the map is not linear in h. The nonlinearity is simultaneously the adapter’s source of power and the root of its inference-latency cost, a trade we quantify below.

Near-identity initialization and why training is stable

Adapters train stably because they start as (near) identity functions. Initialize W_up = 0 (or with very small variance) and at step zero the adapter outputs Adapter(h) = h + 0 = h: the modified network computes exactly what the pretrained network computed. Training then perturbs the model continuously away from a known-good solution instead of injecting random noise into the middle of every layer, which would wreck the pretrained features before learning begins.

The gradient math shows why this works and does not stall. For the up-projection, ∂L/∂W_up = δ · f(W_down h)^T, which is generically nonzero at initialization, so W_up moves immediately. For the down-projection, ∂L/∂W_down carries a factor of W_up^T and is zero at step zero — but after one update to W_up it becomes nonzero, and both matrices co-train. LoRA later copied this recipe exactly (B = 0, A random); zero-init of the output-side matrix is the shared trick that makes all residual PEFT modules safe to insert.

Placement: Houlsby vs Pfeiffer

Where the adapter sits inside the transformer block matters nearly as much as its size. The original Houlsby configuration inserts two adapters per block: one after the multi-head attention sublayer and one after the feed-forward (FFN) sublayer, each applied after the sublayer’s projection and before (or around) the residual add and LayerNorm. Two insertion points give the most adaptation capacity per block.

Pfeiffer et al. (2020) showed via ablation that most of the benefit comes from a single adapter placed after the FFN sublayer only — the Pfeiffer configuration — halving adapter parameters and compute at almost no accuracy cost. The intuition: the FFN output is the block’s final write into the residual stream, so correcting there catches everything the block contributes. In both schemes the LayerNorm parameters (2d per norm) are usually unfrozen too, since they are cheap and rescaling statistics helps on shifted domains. Modern libraries (AdapterHub / adapters, HF PEFT) expose both layouts as configs.

Serial vs parallel adapters

The classic adapter is serial: it sits in the data path, transforming the sublayer output before it continues, so its latency adds directly to the critical path. He et al. (2021, “Towards a Unified View of PEFT”) proposed the parallel adapter: run the bottleneck alongside the sublayer on the same input, and add both outputs into the residual stream:

serial:    out = h + FFN(h);  out′ = out + W_up f(W_down out)
parallel:  out = h + FFN(h) + s · W_up f(W_down h)

with a scaling factor s (often 2–4). Empirically the parallel form matches or beats serial at equal parameter count, and it exposes the deep connection to LoRA: a parallel adapter on a weight’s input/output, with the nonlinearity removed, is LoRA. The unified view reads all these methods as one family — “add a learned low-dimensional delta to the residual stream” — differing only in placement, linearity, and merge-ability.

Counting the parameters exactly

Per adapter, the parameter count is the two matrices plus biases:

p_adapter = (d×r + r)  +  (r×d + d)  =  2rd + r + d

For a model with L layers and k adapters per block (Houlsby k=2, Pfeiffer k=1), total trainable parameters are L · k · (2rd + r + d), plus 2d per unfrozen LayerNorm and the task head. Note the count is linear in r: doubling the bottleneck doubles the adapter, so r is a clean capacity dial. Typical values run r = d/16 down to r = d/64; Houlsby’s sweep found accuracy degrades gracefully as r shrinks, with even r = 8 competitive on many GLUE tasks.

Compare LoRA on a single d × d weight: 2rd parameters, essentially identical. At equal r the two methods cost the same to store; they differ in where the capacity sits and what happens at inference.

A worked example: adapters on a 125M CPU SLM

Take a small language model with d = 768, L = 12 layers, ~125M parameters — a typical CPU-class SLM. Choose Pfeiffer placement (k = 1) with bottleneck r = 48 = d/16:

per adapter: 2rd + r + d = 2·48·768 + 48 + 768
           = 73,728 + 816 = 74,544 params
12 layers:  12 × 74,544        ≈ 0.894M
+ LayerNorms: 12 × 2 × 2·768  ≈ 0.037M
total ≈ 0.93M trainable  =  0.75% of 125M

Each task checkpoint is ~0.93M weights ≈ 1.9 MB in fp16 — versus ~250 MB for a full fine-tuned copy. Ten tasks cost 19 MB on top of one shared backbone instead of 2.5 GB. Training memory falls too: Adam moments exist only for 0.93M weights (~11 MB in fp32) instead of 1.5 GB, which is exactly what makes fine-tuning feasible on a laptop-class CPU with 16 GB of RAM.

What adapters cost at inference

The forward FLOPs of one adapter per token are two thin matmuls, roughly 2 · (2rd) = 4rd multiply-adds. Against the block’s FFN at 2 · (2 · d · 4d) = 16d², the ratio is 4rd / 16d² = r/4d — for r = d/16 that is ~1.6% extra compute. On paper, negligible.

In practice the cost is larger than the FLOP count suggests, for two reasons. First, the adapter is an extra sequential step on the critical path: two more (small) GEMMs, an activation, and a residual add per adapted sublayer, per layer. In memory-bandwidth-bound CPU decoding at batch size 1, each extra op pays kernel-launch and cache-traffic overhead disproportionate to its FLOPs; measured overheads of 5–10% per token are common for Houlsby-style stacks. Second, the matrices are skinny ([768, 48]), a shape at which GEMM efficiency is poor — the hardware is latency-bound, not throughput-bound. Adapters are cheap, but they are permanently cheap: the cost recurs on every token, forever.

The merging asymmetry: adapters vs LoRA

This is the decisive structural difference. A LoRA update is linear: W′ = W + (α/r) BA can be computed once, after which the deployed model has exactly its original architecture, layer count, and per-token latency — zero overhead. An adapter’s nonlinearity makes the equivalent fold impossible: h + W_up f(W_down h) is not expressible as any fixed matrix times h, so the module must execute at every step of inference.

The flip side is that merging destroys modularity. A merged LoRA serves one task; switching tasks means re-patching weights (cheap, but stateful and exclusive). Unmerged adapters keep the backbone pristine and make the task a runtime argument: one resident model, many 2 MB adapter sets, switchable per request. AdapterHub built an ecosystem on exactly this — download a task adapter like a plugin. The right mental model: LoRA optimizes single-task latency; adapters optimize multi-task density. (LoRA can also run unmerged to regain modularity — at which point its overhead story matches the adapter’s.)

Choosing r, and other practical pitfalls

Common failure modes are worth naming. Oversizing r: accuracy vs r saturates quickly; past the knee you buy checkpoint size and latency for nothing. Start at r = d/16 and halve until the dev metric moves. Undersizing on hard shifts: for genuinely new domains (code, legal, another language) the low-dimensional-correction assumption weakens; adapters underperform full fine-tuning more there, and raising r or adapting embeddings helps. Freezing LayerNorm: the 2d norm parameters are nearly free and consistently help; leaving them frozen is a silent accuracy leak.

Learning rate: adapters want higher LRs than full fine-tuning — around 1e-4 to 1e-3 rather than 2e-5 — because the trainable subspace is tiny and zero-initialized. Stacking naively: composing several serial adapters multiplies critical-path latency; if you need composition, prefer parallel placement or fusion techniques (e.g. AdapterFusion) designed for it.

Where adapters fit in the PEFT family

A compact map of the trade space:

Adapters occupy the “expressive but resident” corner: the only method in the table that computes a genuinely nonlinear, input-dependent correction, at the price of living permanently in the forward pass. For a CPU SLM serving many tasks from one frozen backbone — a support bot with per-customer adapters, a router with per-skill modules — that corner is often exactly the right one. For a single-task, latency-critical deployment, merged LoRA wins and it is not close.