The three mechanisms, and the one premise that joins them

QLoRA layers three separable tricks on a single premise: the base weights W are frozen, and all adaptation lives in a low-rank term, so an adapted linear layer computes

y = dequant(W_q) · x  +  (α/r) · B · A · x

W_q: [d_out, d_in] stored in NF4 (4 bits/weight, frozen)
A:   [r, d_in]     bf16, trainable
B:   [d_out, r]    bf16, trainable, initialized to 0

NF4 chooses which sixteen values a 4-bit code may represent. Double quantization compresses the per-block scale factors NF4 needs. Paged optimizers keep the run alive when allocations spike. The three are orthogonal, each attacking a different line of the memory bill, and the theory worth understanding is why the first two cost almost no quality — an answer that comes from the premise, not from the quantizer.

Advertisement

NF4: a grid matched to the data

A 4-bit code has sixteen slots. Uniform int4 spreads them evenly across [-absmax, +absmax], which is right only if the weights are uniformly distributed. They are not: after block-wise normalization, transformer weights are close to zero-centered Gaussian, so a uniform grid spends slots on empty tails and starves the dense region near zero.

NF4 places its sixteen levels at the quantiles of a standard normal, rescaled to [-1, 1], with an asymmetric split so that exact 0 is representable — padding and genuinely-zero weights then quantize with no error. Each bin carries roughly equal probability mass, which minimizes expected quantization error for Gaussian data. The usual claim that this is information-theoretically optimal holds only under the normality assumption and the equal-mass objective; learned or slightly perturbed grids can edge it out. Treat it as an excellent prior, not a theorem about your weights.

Advertisement

Double quantization: the bits hiding in the scale factors

NF4 is applied per block of 64 weights, and each block needs its own absmax constant to map codes back to real magnitudes. Those constants are not free, and the arithmetic is easy to do yourself:

one FP32 constant per 64 weights   →  32/64        = 0.500 bits/param

double quantization:
  quantize constants to 8-bit    →  8/64         = 0.125
  one FP32 constant per 256 of   →  32/(64*256)  = 0.002
                                     ----------------------
                                     total        = 0.127 bits/param

saving = 0.500 - 0.127 ≈ 0.37 bits/param
effective footprint = 4 + 0.127 = 4.127 bits/param

Roughly 0.37 bits per parameter sounds negligible until you multiply: on a 65B model that is 0.37 × 65e9 / 8 ≈ 3.0 GB — the difference between fitting on one 48 GB accelerator and not. Double quantization is pure bookkeeping: it touches metadata, never the weights, so its quality cost is essentially zero.

Storage dtype is not compute dtype

The most common misreading of QLoRA is that the matmul happens in 4 bits. It does not. NF4 is a storage format; the arithmetic runs in a compute dtype, normally bf16. At each layer the kernel dequantizes one block of weights at a time, multiplies in bf16, and discards the dequantized values. The full 16-bit model is never materialized anywhere.

Two consequences follow. First, numerical quality is set by the compute dtype, not the storage dtype: accumulation is as accurate as in any bf16 LoRA run, and what changed is only that the operand W is a lossy reconstruction. Second, QLoRA is slower per step than 16-bit LoRA, not faster — you have added a dequantization stage to every weight read. What you bought is capacity: steps you could not run at any speed become steps you can run slowly. Trading throughput for feasibility is the entire deal.

The gradient argument: which tensors exist and which never do

Here is the structural reason quantizing the base is cheap. Write the backward pass for the adapted layer explicitly, with g = ∂L/∂y:

∂L/∂x = dequant(W_q)^T · g  +  (α/r) · A^T · B^T · g
∂L/∂A = (α/r) · B^T · g · x^T        shape [r, d_in]
∂L/∂B = (α/r) · g · (A · x)^T        shape [d_out, r]

∂L/∂W  —  never computed, never allocated

The frozen weights appear in exactly one place: as the transpose operator that propagates ∂L/∂x backwards. No tensor of shape [d_out, d_in] is allocated for a gradient, and no optimizer moments are kept for it — gradients flow through the quantized base, never accumulate into it. The gradients that do exist, for A and B, are computed in bf16 from bf16 parameters. The trainable path is at full LoRA precision end to end; only the constant it is measured against was rounded.

A worked memory ledger

Take a 7B model (d = 4096, ffn 11008, 32 layers), LoRA rank r = 64 on all seven linear projections per block:

adapter params/layer = 4 × 64 × (4096+4096)  = 2.10M   (q,k,v,o)
                     + 3 × 64 × (4096+11008) = 2.90M   (gate,up,down)
                     = 5.0M  × 32 layers      ≈ 160M params

NF4 base   : 7.0e9 × 4.127 bits / 8   ≈ 3.6 GB
adapters   : 160e6 × 2 B (bf16)       ≈ 0.32 GB
adapter grads                            ≈ 0.32 GB
Adam m,v in fp32 : 160e6 × 8 B        ≈ 1.28 GB
                                          --------
                                          ≈ 5.5 GB + activations

Compare 16-bit full fine-tuning of the same model: 14 GB of weights, 14 GB of gradients, and 56 GB of fp32 Adam moments, before activations. The dominant saving is not the 4-bit base at all; it is the 56 GB of optimizer state that ceases to exist when 99 % of parameters are frozen. Quantization is what makes the remaining frozen block small enough to sit alongside it.

Why the error does not degrade adaptation — and why coverage matters

QLoRA matches 16-bit fine-tuning quality, while quantizing a 16-bit fine-tuned model afterwards visibly does not. Write the quantized base as W̄ = W + E, with E the rounding error. During QLoRA training E is fixed: decided once, before the first step, and never changed. The optimizer is not fighting noise — it is adapting a slightly different function, and every gradient already accounts for E, so the adapter can absorb part of it, learning a BA that is partly correction and partly task. Post-hoc quantization reverses the order: fine-tune against W, then perturb by E with no capacity left to compensate. Same perturbation, opposite outcome, purely because of where it lands relative to the optimization.

That picture also explains a practical rule. Classic LoRA adapts only W_Q and W_V; QLoRA needs adapters on every linear layer — four attention projections plus three MLP projections — and empirically coverage beats rank. If E is spread across all weight matrices but adapters sit on two, the corrective capacity is concentrated where the error is not. Coverage buys a correction term everywhere the perturbation lives; rank only deepens one you already have.