Full fine-tuning updates every weight in the model; parameter-efficient fine-tuning (PEFT) freezes the base and trains a tiny set of new parameters — LoRA low-rank updates, adapters, or bias terms. The sibling articles derive how each PEFT method works; this one is the decision. The honest question is not ‘which is more elegant’ but ‘which do I run, on what hardware, and at what quality?’ The answer turns almost entirely on arithmetic: the memory a training step costs, the storage a hundred fine-tuned variants occupy, and how far a low-rank patch can move a frozen network before it stops keeping up with a full update. This piece works those numbers on a concrete 7B model, separates the memory win from the compute win (they are not the same size), pins down where the quality gap really opens, and lays out when full fine-tuning is still the right call.
What each side actually changes
Start with the object being optimized. A transformer with P parameters defines a loss surface in P dimensions. Full fine-tuning lets gradient descent move in all of them: every attention projection, every feed-forward matrix, every embedding is a free variable. PEFT fixes the base weights at their pretrained values and opens only a small subspace of new parameters — for LoRA, a rank-r update ΔW = B A added to chosen matrices, with A: [r, d] and B: [d, r].
That single distinction — all P dimensions free versus a frozen base plus a thin trainable slice — propagates into every practical difference that follows: how much optimizer state you carry, how big a checkpoint is, how many tasks one GPU can serve, and how much the model can forget. Everything below is a consequence of where the free parameters live.
The memory bill of full fine-tuning
Training memory is dominated not by the weights but by what you keep alongside them. The standard mixed-precision recipe with Adam carries, per trainable parameter: an fp16 weight (2 bytes), an fp16 gradient (2), an fp32 master copy of the weight (4), and Adam’s two moments m and v in fp32 (4 + 4). That is the well-known 16 bytes per parameter before a single activation is stored.
The consequence is brutal for full FT, because every parameter is trainable. A 7-billion-parameter model needs 7e9 × 16 ≈ 112 GB just for weights, gradients, and optimizer state — before activations, which add more and scale with batch size and sequence length. That already overflows a single 80 GB accelerator, which is why full fine-tuning of even mid-sized models pulls in sharded optimizers, ZeRO, or multi-GPU setups. The 16-bytes rule is what makes full FT expensive.
Why freezing the base collapses the bill
PEFT attacks exactly the term that dominates. Freeze the base and those 7 billion parameters need no gradient, no fp32 master copy, and no optimizer moments — they contribute nothing to the 16-bytes column. They still have to be resident for the forward and backward passes, but resident is cheap: 2 bytes each in fp16, so about 14 GB. The full 16-byte treatment applies only to the handful of trainable adapter parameters.
For a 7B model, a typical LoRA configuration (rank 16 on the attention projections) trains on the order of 10 million parameters — well under 0.2% of the model. Their optimizer footprint, 1e7 × 16 ≈ 160 MB, is a rounding error, so total training memory drops from ~112 GB toward the ~14 GB of the resident frozen weights. QLoRA pushes further, storing the frozen base in 4-bit (~0.5 bytes/param, ~3.5 GB for 7B) so the whole job fits a single consumer card.
A worked memory comparison
Putting the two side by side for a 7B model under mixed-precision Adam, before activations:
| Component | Full FT | LoRA (r=16) |
|---|---|---|
| Resident weights (fp16) | 14 GB | 14 GB (frozen) |
| fp32 master copy | 28 GB | — |
| Gradients | 14 GB | ~0.02 GB |
| Adam m + v (fp32) | 56 GB | ~0.13 GB |
| Trainable params | 7 B | ~10 M |
| Total (ex-activations) | ~112 GB | ~14 GB |
The eightfold gap is the entire reason PEFT exists — the difference between an eight-GPU node and a single 24 GB card. Note that the base weights are counted once, not per task — the serving section turns that into the second big win.
Storage: one checkpoint versus many deltas
Memory is the training-time cost; storage is the lifetime cost, and here the gap is even wider. A full fine-tune produces a complete new set of weights. For a 7B model that is a ~14 GB checkpoint in fp16 — per task, per experiment, per checkpoint you keep. Ten variants are 140 GB; a hundred exceed a terabyte.
PEFT stores only the delta. A LoRA adapter for that same model is the A and B matrices alone — typically tens of megabytes, sometimes less. A hundred LoRA adapters might total a few gigabytes, and they all share the one base checkpoint you already have. This changes what is feasible: you can keep a per-customer or per-domain adapter library, version it cheaply, and ship a new capability as a small file rather than a full model release. When the plan is many specializations of one base, storage alone often decides the question before quality is even discussed.
Is PEFT actually faster per step?
This is the most common misconception, so be precise. PEFT’s giant win is memory and storage, not raw per-step compute. The forward pass must still run the entire frozen network — freezing a weight does not make its matmul disappear — plus a small extra cost for the adapter’s x A B product. So the forward FLOPs are essentially unchanged, or marginally higher.
The backward pass is where a modest saving appears. Backprop through a linear layer computes two things: the gradient w.r.t. its input (needed to keep propagating) and the gradient w.r.t. its weight. Frozen layers skip the second, trimming roughly a third of their backward cost, but they must still compute the first to reach the adapters below — so activations for the full network are still needed and mostly still stored. The upshot: per-step wall time drops somewhat, but the transformational effect is that the smaller memory budget lets you use a bigger batch or cheaper hardware, which is where the real throughput and cost wins come from.
The quality gap, honestly
If PEFT were free and lossless, full fine-tuning would be a historical footnote. It is not quite lossless. On a large class of downstream tasks — classification, instruction-style adaptation, style and format shaping, moderate domain specialization — a well-tuned LoRA lands within about a point of full fine-tuning, close enough that the memory and storage savings dominate the decision.
The gap widens in specific regimes. When the task demands genuinely new knowledge or a large distribution shift — a new language, a specialized corpus with unfamiliar vocabulary, a capability the base lacks — a low-rank patch can hit a ceiling that full fine-tuning clears. Rank is the knob: too small and LoRA underfits; raising r recovers capacity at the cost of the efficiency advantage. Counter-intuitively, on small datasets PEFT can generalize better, because freezing most weights is a strong regularizer that keeps a full fine-tune from overfitting away the base model’s knowledge.
Multi-task serving: PEFT's decisive edge
The comparison flips from ‘nice to have’ to ‘no contest’ the moment you must serve many specializations at once. With full fine-tuning, each task is an independent 14 GB model. Serving N tasks means N × 14 GB resident in GPU memory, or paying to load and unload full checkpoints per request — both brutal past a handful of tasks.
PEFT collapses this. One frozen base sits in memory once; each task is a small adapter you swap in. Serving a hundred specializations costs 14 GB + 100 × (a few MB), still comfortably one GPU. Modern serving systems (the S-LoRA / Punica line of work) go further and batch requests for different adapters together against the shared base, so a single server multiplexes thousands of adapters at near the throughput of serving one. There is no full-fine-tuning equivalent. If your product is ‘one model, many tenants or personas,’ PEFT is the architecture, not merely the cheaper training trick.
Reversibility and catastrophic forgetting
Full fine-tuning writes over the pretrained weights, and with them some of the general capability that made the base valuable. Push hard on a narrow task and the model can degrade elsewhere — the catastrophic forgetting and alignment-tax problem — and the only way back is a checkpoint you saved beforehand.
PEFT is structurally safer here because the base is literally untouched. The original behavior is one ΔW subtraction away: detach the adapter and you have the exact starting model back, bit for bit. That makes specializations composable and disposable — test an adapter, roll it back, swap another — and it bounds the blast radius of a bad fine-tune to the adapter rather than the whole model. For safety-sensitive or audited deployments, that reversibility and the preservation of the base model’s vetted behavior are worth real quality points, quite apart from the memory math — full FT offers no such undo button short of reloading gigabytes.
When full fine-tuning still wins
PEFT is the default for most task adaptation, but full fine-tuning earns its keep in clear cases. Reach for it when you are doing continued pretraining or absorbing a large new corpus, where the goal is to move the base distribution itself rather than patch a task on top. Reach for it when the domain shift is large enough that a low-rank update demonstrably underfits and raising rank has erased the efficiency edge anyway.
It also wins when the economics invert: a single high-value model served at massive scale, where you will happily spend the training compute once to squeeze out the last point of quality, and where the multi-task serving argument never applies because there is only one task. The honest summary: full FT wins when you are changing the model deeply, or when there is exactly one model and quality is the only axis that matters.
A decision framework
Strip it to the questions that actually decide the call:
| Ask… | Lean PEFT if… |
|---|---|
| How many specializations to serve? | More than one — shared base, swappable adapters |
| How much GPU memory? | Limited — PEFT makes it a single-card job |
| How far is the task from pretraining? | Near — a low-rank patch keeps up |
| Is the dataset small? | Yes — the frozen base regularizes |
| Need reversibility / base preservation? | Yes — detach to restore the exact base |
| Moving the base distribution itself? | No — if yes, that is full FT |
The rows rarely disagree in practice. For most teams adapting a capable base to specific tasks, PEFT is the sensible default and full fine-tuning the deliberate exception — reserved for deep changes to the model, or a single model where quality is all that matters.