QLoRA is not one idea but three that happen to compose: a base model stored in 4-bit NF4, a second pass of quantization applied to the quantization constants themselves, and a paged optimizer that survives memory spikes — with an ordinary 16-bit LoRA adapter trained on top. What makes the combination interesting is not the compression ratio. It is that the compression sits on the frozen half of the model, where the backward pass needs the weights only as a linear operator and never as a thing to differentiate. That one structural fact is why you can throw away three quarters of the bits in the base and still recover 16-bit fine-tuning quality.
The three mechanisms, and the one premise that joins them
QLoRA layers three separable tricks on a single premise: the base weights W are frozen, and all adaptation lives in a low-rank term, so an adapted linear layer computes
y = dequant(W_q) · x + (α/r) · B · A · x
W_q: [d_out, d_in] stored in NF4 (4 bits/weight, frozen)
A: [r, d_in] bf16, trainable
B: [d_out, r] bf16, trainable, initialized to 0NF4 chooses which sixteen values a 4-bit code may represent. Double quantization compresses the per-block scale factors NF4 needs. Paged optimizers keep the run alive when allocations spike. The three are orthogonal, each attacking a different line of the memory bill, and the theory worth understanding is why the first two cost almost no quality — an answer that comes from the premise, not from the quantizer.
NF4: a grid matched to the data
A 4-bit code has sixteen slots. Uniform int4 spreads them evenly across [-absmax, +absmax], which is right only if the weights are uniformly distributed. They are not: after block-wise normalization, transformer weights are close to zero-centered Gaussian, so a uniform grid spends slots on empty tails and starves the dense region near zero.
NF4 places its sixteen levels at the quantiles of a standard normal, rescaled to [-1, 1], with an asymmetric split so that exact 0 is representable — padding and genuinely-zero weights then quantize with no error. Each bin carries roughly equal probability mass, which minimizes expected quantization error for Gaussian data. The usual claim that this is information-theoretically optimal holds only under the normality assumption and the equal-mass objective; learned or slightly perturbed grids can edge it out. Treat it as an excellent prior, not a theorem about your weights.
Double quantization: the bits hiding in the scale factors
NF4 is applied per block of 64 weights, and each block needs its own absmax constant to map codes back to real magnitudes. Those constants are not free, and the arithmetic is easy to do yourself:
one FP32 constant per 64 weights → 32/64 = 0.500 bits/param
double quantization:
quantize constants to 8-bit → 8/64 = 0.125
one FP32 constant per 256 of → 32/(64*256) = 0.002
----------------------
total = 0.127 bits/param
saving = 0.500 - 0.127 ≈ 0.37 bits/param
effective footprint = 4 + 0.127 = 4.127 bits/paramRoughly 0.37 bits per parameter sounds negligible until you multiply: on a 65B model that is 0.37 × 65e9 / 8 ≈ 3.0 GB — the difference between fitting on one 48 GB accelerator and not. Double quantization is pure bookkeeping: it touches metadata, never the weights, so its quality cost is essentially zero.
Storage dtype is not compute dtype
The most common misreading of QLoRA is that the matmul happens in 4 bits. It does not. NF4 is a storage format; the arithmetic runs in a compute dtype, normally bf16. At each layer the kernel dequantizes one block of weights at a time, multiplies in bf16, and discards the dequantized values. The full 16-bit model is never materialized anywhere.
Two consequences follow. First, numerical quality is set by the compute dtype, not the storage dtype: accumulation is as accurate as in any bf16 LoRA run, and what changed is only that the operand W is a lossy reconstruction. Second, QLoRA is slower per step than 16-bit LoRA, not faster — you have added a dequantization stage to every weight read. What you bought is capacity: steps you could not run at any speed become steps you can run slowly. Trading throughput for feasibility is the entire deal.
The gradient argument: which tensors exist and which never do
Here is the structural reason quantizing the base is cheap. Write the backward pass for the adapted layer explicitly, with g = ∂L/∂y:
∂L/∂x = dequant(W_q)^T · g + (α/r) · A^T · B^T · g
∂L/∂A = (α/r) · B^T · g · x^T shape [r, d_in]
∂L/∂B = (α/r) · g · (A · x)^T shape [d_out, r]
∂L/∂W — never computed, never allocatedThe frozen weights appear in exactly one place: as the transpose operator that propagates ∂L/∂x backwards. No tensor of shape [d_out, d_in] is allocated for a gradient, and no optimizer moments are kept for it — gradients flow through the quantized base, never accumulate into it. The gradients that do exist, for A and B, are computed in bf16 from bf16 parameters. The trainable path is at full LoRA precision end to end; only the constant it is measured against was rounded.
A worked memory ledger
Take a 7B model (d = 4096, ffn 11008, 32 layers), LoRA rank r = 64 on all seven linear projections per block:
adapter params/layer = 4 × 64 × (4096+4096) = 2.10M (q,k,v,o)
+ 3 × 64 × (4096+11008) = 2.90M (gate,up,down)
= 5.0M × 32 layers ≈ 160M params
NF4 base : 7.0e9 × 4.127 bits / 8 ≈ 3.6 GB
adapters : 160e6 × 2 B (bf16) ≈ 0.32 GB
adapter grads ≈ 0.32 GB
Adam m,v in fp32 : 160e6 × 8 B ≈ 1.28 GB
--------
≈ 5.5 GB + activationsCompare 16-bit full fine-tuning of the same model: 14 GB of weights, 14 GB of gradients, and 56 GB of fp32 Adam moments, before activations. The dominant saving is not the 4-bit base at all; it is the 56 GB of optimizer state that ceases to exist when 99 % of parameters are frozen. Quantization is what makes the remaining frozen block small enough to sit alongside it.
Why the error does not degrade adaptation — and why coverage matters
QLoRA matches 16-bit fine-tuning quality, while quantizing a 16-bit fine-tuned model afterwards visibly does not. Write the quantized base as W̄ = W + E, with E the rounding error. During QLoRA training E is fixed: decided once, before the first step, and never changed. The optimizer is not fighting noise — it is adapting a slightly different function, and every gradient already accounts for E, so the adapter can absorb part of it, learning a BA that is partly correction and partly task. Post-hoc quantization reverses the order: fine-tune against W, then perturb by E with no capacity left to compensate. Same perturbation, opposite outcome, purely because of where it lands relative to the optimization.
That picture also explains a practical rule. Classic LoRA adapts only W_Q and W_V; QLoRA needs adapters on every linear layer — four attention projections plus three MLP projections — and empirically coverage beats rank. If E is spread across all weight matrices but adapters sit on two, the corrective capacity is concentrated where the error is not. Coverage buys a correction term everywhere the perturbation lives; rank only deepens one you already have.
Paged optimizers: what they are, and what they are not
Even with 5.5 GB of steady-state usage, a run can die on a single bad step: a long sequence, a gradient-checkpointing recomputation, an unlucky allocator fragment. Paged optimizers address exactly this. Optimizer state is allocated in CUDA unified memory, so when the device is under pressure the driver evicts those pages to host RAM and faults them back when the update step touches them — OS virtual memory, applied to accelerator allocations.
Be precise about what this is. It is not compression: the bytes still exist, merely somewhere else. It does not lower average memory use; it raises the ceiling on transient spikes, at the price of PCIe traffic on the steps that page. And it is CUDA-specific with no CPU-side analogue — on a CPU box the operating system already pages your memory, so the honest equivalent is bounding the peak directly with smaller micro-batches and shorter sequences.
The merge asymmetry: you cannot fold BA into NF4
Standard LoRA has a clean endgame: compute W’ = W + (α/r)BA once, ship a single matrix, pay zero inference overhead. With a 4-bit base that path is not exact. Merging requires arithmetic and NF4 supports none, so you must dequantize first: W’ = dequant(W_q) + (α/r)BA, a bf16 matrix at full size.
If you then re-quantize, quant(W’) ≠ W_q + quant((α/r)BA) — a fresh rounding error lands on the adapted weights, and unlike the training-time error it arrives after optimization, so nothing can absorb it. You are back in the post-hoc regime the method was designed to avoid. Three honest options: keep the adapter separate at inference (exact, small latency cost), merge and serve in 16-bit (exact, full memory), or re-quantize and measure the damage. What you cannot do is assume merging is free.
What transfers to a CPU SLM, and the usual pitfalls
QLoRA’s value is proportional to how badly the base does not fit. For a 125M–1.5B CPU SLM the 16-bit base is 0.25–3 GB and fits in ordinary system RAM, so plain bf16 LoRA is usually the better trade: same adapter memory, no per-step dequantization tax, no merge asymmetry at the end. Reach for QLoRA when the base genuinely will not fit, not as a default.
The pitfalls that actually bite: expecting a speedup (you will get a slowdown); leaving double quantization off and paying an unnecessary 0.37 bits per parameter; setting the compute dtype to fp32 and losing the bandwidth advantage; adapting only q/v and then blaming the quantizer for the quality gap; and evaluating a merged-then-re-quantized checkpoint while reporting it as the QLoRA result.