bitsandbytes is two things people conflate. One is LLM.int8(), the mixed-precision matmul algorithm for inference — that is a separate story. The other, the subject here, is the library’s underlying 8-bit number formats and the 8-bit optimizers built on them. The engine is block-wise absmax quantization: chop a tensor into small blocks, give each block its own scale, and a single wild outlier can only ruin the block it lives in. Layer a non-linear dynamic code on top and you get high precision for the near-zero values that dominate weights, gradients, and Adam state alike. Put those together and you can store Adam’s two 32-bit moment buffers in 8 bits each — cutting optimizer memory by 75% with almost no accuracy cost. This piece walks the absmax primitive, the block-wise trick and why it works, linear vs dynamic formats, 8-bit optimizers, a worked memory example, and where it bites on a small CPU box.
The absmax int8 primitive
Everything starts with one idea: map a range of floats onto the 256 integers an 8-bit signed value can hold. bitsandbytes uses absmax quantization. For a block of values b, take the largest magnitude a = max_i |b_i|, form a scale c = 127 / a, and store q_i = round(c · b_i) as an int8 in [-127, 127]. To read a value back you divide out the scale: b_i ≈ q_i / c.
c = 127 / max_i |b_i| (one fp16/fp32 scale)
q_i = round(c * b_i) (int8, in [-127, 127])
b_i ~= q_i / c (dequantize)The format is symmetric (zero maps to zero, no separate zero-point) and uniform: the 255 levels are evenly spaced across [-a, +a]. The worst-case rounding error per value is half a step, a / 254 ≈ a/255. That single fact is the whole tension of the method: the error floor is set by the block’s largest element, so anything that inflates a without raising the typical value degrades every value in the block.
Why one big outlier wrecks a whole-tensor scale
Suppose you quantize an entire weight matrix with a single absmax. Neural network tensors are famously heavy-tailed: most values sit within, say, [-1, 1], but a handful of outlier features spike to 30 or 100. Absmax is driven by that maximum. If a = 100, the step size is 100/127 ≈ 0.79 — so every ordinary weight near 0.3 rounds to 0 or ±1. The bulk of the tensor collapses onto a few integer levels and its precision is destroyed, all to faithfully represent a couple of extreme values.
This is not a corner case; it is the normal behavior of transformer activations and of optimizer state during training. A per-tensor scale forces the common mass and the rare tail to share one dynamic range, and the tail always wins. You could clip the outliers, but clipping throws away information that often matters. The better move is to stop letting one outlier speak for the whole tensor — which is exactly what block-wise quantization does.
Block-wise absmax: a scale per block
Block-wise quantization flattens the tensor and splits it into contiguous blocks of fixed size B (bitsandbytes uses B = 2048 for optimizer states, and similar sizes elsewhere). Each block computes its own absmax a_k = max_{i in block k} |b_i| and is quantized independently against it. Instead of one scale for N values you now keep N / B scales, one per block.
tensor -> [ block_0 | block_1 | ... | block_{N/B-1} ]
each block k: a_k = max|b|, c_k = 127/a_k, q = round(c_k * b)
store: N int8 codes + (N/B) fp32 absmax scalesBecause every block is normalized against its own maximum, the blocks are independent: they can be quantized and dequantized fully in parallel, which maps cleanly onto GPU threads and is cheap on CPU too. The cost is the extra scales, but they are tiny (quantified below). The payoff is that the dynamic range is now local — and locality is what tames the tail.
Why blocks bound outlier damage
Here is the key argument. An outlier inflates the absmax of one block of B values, not the whole tensor. With N = 4,194,304 weights and B = 2048, a lone spike corrupts the resolution of at most 2048 values — roughly 0.05% of the tensor. The other 2047 blocks never see that outlier; each keeps its own tight absmax (maybe a_k ≈ 1) and therefore a fine step of 1/127 ≈ 0.008, preserving full int8 resolution for the bulk of the data.
Contrast the two error floors directly. Whole-tensor: every value pays a_global / 254, dragged up by the global max. Block-wise: a value pays only a_k / 254, set by its neighbors. Since outliers are sparse and clustered, almost every block has a small local max, so almost every value gets a small error. Block-wise quantization does not remove outliers — it quarantines them, capping the blast radius of each spike to a single block. That is why it is robust enough to quantize quantities that a per-tensor scheme cannot touch.
Linear vs dynamic 8-bit formats
Absmax fixes the range; the format decides how the 255 codes are spaced inside it. bitsandbytes offers two. The linear format spaces levels uniformly — simple, and fine when values fill the range evenly. But weights, gradients, and Adam moments are concentrated near zero with a long tail, and uniform spacing wastes most of its codes on large magnitudes that rarely occur while starving the crowded region near zero.
The dynamic format fixes this with a non-linear, dynamic-exponent code. Each 8-bit value splits its bits between an exponent (how big) and a fraction (how precise), and the split floats: small numbers spend more bits on the exponent to get many closely spaced levels near zero, while large numbers trade fraction bits for reach. The result is high relative precision across many orders of magnitude from a single byte — a small floating-point-like code rather than a fixed grid. Paired with block-wise normalization, dynamic quantization is what lets 8 bits stand in for 32-bit optimizer state without derailing training.
8-bit optimizers: the memory Adam hides
Adam’s memory cost is easy to underestimate. For every parameter it keeps two 32-bit buffers: the first moment m (the running mean of the gradient) and the second moment v (the running mean of the squared gradient). At 4 bytes each that is 8 bytes of optimizer state per parameter — twice the size of the fp32 model itself. For large models this state, not the weights, is what overflows memory.
8-bit optimizers store m and v as block-wise, dynamically quantized int8 — 1 byte each instead of 4. Crucially the update math stays in 32-bit. On each step the relevant block is dequantized to fp32, the standard Adam update is applied, and the fresh state is re-quantized back to 8-bit for storage. So the moving averages accumulate at full precision moment to moment; only the resting representation is compressed. bitsandbytes ships drop-in classes (Adam8bit, AdamW8bit, and others) that swap in for the 32-bit versions, plus a stable embedding layer that normalizes the embedding — a known instability source — to keep large-scale training well-behaved.
A worked memory-savings example
Take a 1.5B-parameter small language model you want to fine-tune. Count the fp32 training footprint, ignoring activations:
P = 1.5e9 params
weights (fp32) : 4 * P = 6.0 GB
gradients (fp32) : 4 * P = 6.0 GB
Adam m (fp32) : 4 * P = 6.0 GB \
Adam v (fp32) : 4 * P = 6.0 GB / optimizer state = 12.0 GB
-------------------------------------------------
total : 24.0 GBNow switch to an 8-bit optimizer. Each moment drops from 4 bytes to 1: m and v become 1.5 GB each, so optimizer state falls from 12.0 GB to 3.0 GB — a 9 GB saving, a 75% cut to the state, and total training memory down from 24 GB to 15 GB. The block scales barely register: one fp32 absmax per 2048 values is 4 / 2048 ≈ 0.002 bytes per value, about 0.2% overhead — so each state costs roughly 8.02 bits, not 8. That 9 GB is often the difference between a job that fits and one that does not.
How the format machinery differs from LLM.int8()
It is worth drawing the line cleanly, because both live in the same library. LLM.int8() is an inference matmul algorithm: it quantizes weights and activations vector-wise (a scale per row or column), then detects the rare outlier feature dimensions and computes those in 16-bit while the rest run in int8 — a mixed-precision decomposition to preserve accuracy on a forward pass. That algorithm is the subject of a separate article.
What this article covers is the layer beneath and beside it: the general-purpose block-wise absmax format (quantize_blockwise / dequantize_blockwise) and the dynamic 8-bit code, which are the storage machinery for 8-bit optimizers and quantized buffers. LLM.int8() uses vector-wise scales and a fp16 outlier path; the block-wise format uses per-block scales and a non-linear code and never splits precision by dimension. One is a matmul strategy for serving, the other is a number format for compressing state during training.
On a small CPU box, and the pitfalls
For CPU-hosted SLM work the practical draw is the optimizer. Fine-tuning is usually gated by RAM, and Adam’s 8 bytes per parameter of state is the first thing to blow the budget; halving or quartering it can turn an impossible run into a routine one, letting you fit a larger model or a bigger batch in the same memory. Block-wise quantization is arithmetically cheap too — a max, a multiply, a round per block.
The pitfalls are real, though. The dequantize-update-requantize cycle adds work every step, and bitsandbytes’ fastest kernels are GPU-tuned, so CPU throughput can lag. Block size is a genuine trade-off: smaller blocks isolate outliers better but store more scales; larger blocks are leaner but let a spike reach more values. And do not over-generalize the savings — quantizing optimizer state is safe precisely because the update runs in fp32; quantizing the weights or gradients used in the update itself is a harder problem with different accuracy rules. Match the tool to the buffer.