GGUF is the file format that made local LLMs practical: one self-describing file you can mmap and run on a laptop CPU. But the format is only half the story. The other half is k-quants — llama.cpp’s block-quantization scheme that squeezes a 16-bit weight down to roughly four or five bits with hierarchical, per-sub-block scales, so a 7B model that would need 14 GB in fp16 fits in about 4 GB with little measurable quality loss. This piece opens both the file and the block: how GGUF lays out metadata and tensors for zero-copy loading, how a k-quant super-block stores quantized weights alongside a two-level hierarchy of scales and mins, what the effective bits-per-weight works out to, and why weights are dequantized inside the matmul’s inner loop rather than up front.
From tensors on disk to a running model
A trained model is, concretely, a few hundred named tensors plus the metadata needed to run them: architecture, layer and head counts, the RoPE and normalization settings, and the entire tokenizer. Earlier formats scattered this across a pickle file, a JSON config, a tokenizer blob, and a separate weights file — fine for a Python training rig, awkward for a single C++ binary that wants to load and run in milliseconds.
GGUF (the successor to GGML and GGMF) collapses all of that into one file with a design goal that shapes everything else: the loader should memory-map the file and use tensors with no parsing and no copying. That constraint explains the layout — a typed key/value header, a tensor directory, then raw quantized tensor data laid out contiguously and aligned so the CPU reads it directly. GGUF is how that zero-copy promise is kept; k-quants are what those raw tensor bytes contain.
The container: a typed header, then tensors
A GGUF file has three parts. First a small header: the magic bytes GGUF, a version number (v3 is current), the tensor count, and the metadata key/value count. Second, the metadata — an ordered list of typed key/value pairs. Values are strongly typed (u32, f32, bool, string, nested arrays), so the reader never has to guess. Keys are namespaced strings like llama.attention.head_count, llama.block_count, and the tokenizer.ggml.* family carrying the full vocabulary and merges.
Third comes the tensor directory: for each tensor, its name, dimension count, shape, its ggml_type (the quantization format), and a byte offset into the data region — then the tensor data itself. Because every hyperparameter and the tokenizer travel inside the file, a GGUF is fully self-contained.
Why the layout is mmap-friendly
The offsets in the tensor directory point into a data region where each tensor is padded to an alignment boundary (32 bytes by default, recorded as general.alignment). That alignment is not cosmetic: it lets the quantization kernels issue aligned SIMD loads.
When llama.cpp opens the model it calls mmap. The OS maps the bytes into the process’s address space without reading them; pages are faulted in lazily as the compute touches them. Nothing is deserialized or copied, and the weights are never expanded to fp32 — the in-memory bytes are the on-disk bytes. So startup is near-instant even for a multi-gigabyte model, and several processes mapping the same file share one physical copy in the OS page cache. The compact quantized layout on disk is also the compact layout in RAM — exactly what a memory-bandwidth-bound CPU wants.
Block quantization, the base idea
Quantization stores weights in fewer bits than fp16, but you cannot just round each weight independently — you need a scale to map small integers back to real magnitudes. GGUF applies scales per block rather than per tensor, because a single scale for millions of weights would be swamped by the largest outlier. The legacy Q4_0 type is the clearest example: take 32 weights, find one fp16 scale d, and store each weight as a signed 4-bit integer q in [-8, 7], reconstructed as w = d · q.
The bookkeeping: 2 bytes for d plus 16 bytes of quants = 18 bytes for 32 weights, or 4.5 bits per weight. Add a per-block minimum (Q4_1: w = d · q + m) for an affine fit and it costs a little more. These flat, one-scale-per-block schemes are simple and fast but leave quality on the table. K-quants are the fix.
K-quants: the super-block and hierarchical scales
K-quants (introduced by Iwan Kawrakow in llama.cpp) keep block quantization but add a second level of scaling. A k-quant type groups QK_K = 256 weights into a super-block divided into sub-blocks of 32 or 16 weights. Each sub-block gets its own scale — and, for the affine types, its own minimum — so magnitude variation across the 256 weights is tracked far more finely than a single scale could.
The trick that makes this cheap is that the sub-block scales and mins are themselves quantized, typically to 6 bits, then rescaled by one fp16 super-block scale. So the hierarchy is: a top-level fp16 d (and dmin) for the whole super-block, low-bit sub-block scales/mins beneath it, and the low-bit weight quants at the bottom. You pay a few extra bits for that scale metadata but buy much better reconstruction — why k-quants beat legacy Q4_0-style types at equal size.
A worked Q4_K block layout
Q4_K is the workhorse. Its super-block holds 256 weights as eight sub-blocks of 32, and packs like this:
d : fp16 2 bytes super-block scale for the 8 scales
dmin : fp16 2 bytes super-block scale for the 8 mins
scales : 8 x 6-bit } 12 bytes packed: 16 six-bit values = 96 bits
mins : 8 x 6-bit } (scale_i and min_i per sub-block)
qs : 256 x 4-bit 128 bytes the weight quants (0..15)
----------------------------------------------------------------
total : 144 bytes -> 144 x 8 / 256 = 4.5 bits per weight
dequant, weight j in sub-block i:
w = d * scale_i * q_j - dmin * min_iRead that dequant line carefully: scale_i and min_i are the 6-bit values for the sub-block weight j lives in, and q_j is its 4-bit quant; the fp16 d and dmin lift those integers back to real magnitudes. The effective width, 4.5 bpw, is identical to Q4_0 — but the eight independent affine sub-block fits reconstruct the tensor far more faithfully. That is the whole k-quant bargain in one block.
Q5_K and Q6_K: spending the next bit
The other types vary the same super-block. Q5_K keeps the 256-weight layout and 6-bit sub-block scales/mins but stores 5-bit weight quants: the low 4 bits packed as in Q4_K (128 bytes) plus a 32-byte plane holding each weight’s 5th bit — 176 bytes per super-block, or 5.5 bpw, a clear quality bump for one extra bit.
Q6_K restructures slightly: 256 weights as sixteen sub-blocks of 16, each with a plain int8 scale. It stores the low 4 bits of every weight (128 bytes), the high 2 bits (64 bytes), sixteen int8 scales (16 bytes), and one fp16 d (2 bytes) — 210 bytes, or 6.5625 bpw. The same idea runs down to Q3_K and Q2_K, which lean even harder on the hierarchical scales to stay usable where a flat 2-bit quant would collapse.
Effective bits-per-weight, and a file-size example
Because each type carries fixed scale overhead per fixed-size block, its bits-per-weight is a constant you can compute exactly:
| Type | Block | Bytes/block | Effective bpw |
|---|---|---|---|
| Q2_K | 256 | 84 | 2.625 |
| Q3_K | 256 | 110 | 3.4375 |
| Q4_K | 256 | 144 | 4.5 |
| Q5_K | 256 | 176 | 5.5 |
| Q6_K | 256 | 210 | 6.5625 |
| Q8_0 | 32 | 34 | 8.5 |
Turn that into a real number. A 7-billion-parameter model in fp16 needs 7e9 × 16 / 8 ≈ 14 GB. At a uniform Q4_K (4.5 bpw) they are 7e9 × 4.5 / 8 ≈ 3.9 GB — a 3.5× shrink that turns a data-center model into a laptop model. Q6_K lands near 5.7 GB, Q2_K near 2.3 GB. This predictable scaling is why you can eyeball a GGUF filename and know both its footprint and roughly where it sits on the size-versus-quality curve.
The _S / _M / _L mixtures
Real GGUF filenames rarely say plain Q4_K; they say Q4_K_M or Q5_K_S. The suffix marks a quant mixture, not a different block format. Not all tensors are equally sensitive, so llama.cpp assigns different k-quant types to different tensors within one file. A Q4_K_M build stores most weights as Q4_K but promotes the attention value (wv) and feed-forward down projections — empirically the most error-sensitive matrices — to Q6_K.
So the letters trade a little size for accuracy: _S (small) keeps more tensors at the base width, _M (medium) promotes the sensitive ones, _L promotes more still. A mixture’s effective bpw thus drifts above the base type — Q4_K_M averages roughly 4.8 bpw, not a flat 4.5. Q4_K_M and Q5_K_M are the community default sweet spots: most of the compression, most of the quality.
I-quants and the importance matrix
K-quants choose each block’s scale to minimize plain reconstruction error — treating every weight as equally important. The newer i-quants (the IQ2_XXS, IQ3_XXS, IQ4_NL family) improve on this in two ways. First, they are codebook-based: instead of an evenly spaced integer grid, weights index into a fixed lattice of values tuned to the bell-shaped distribution real weights follow, packing more fidelity into two or three bits.
Second, and more importantly, they are guided by an importance matrix (imatrix). You compute it once by running calibration text through the model and accumulating, per weight column, how much that column drives the activations. Quantization then weights its error budget by importance — spending precision where it changes the output and economizing where it does not. The result: i-quants deliver noticeably better quality than k-quants at the extreme low end (2–3 bits), at the cost of calibration and slower kernels. An imatrix can also sharpen ordinary k-quants.
Dequant in the inner loop
Here is the part that ties the format back to speed. llama.cpp never expands the quantized weights into a full fp16 or fp32 array — not on load, not before the matmul. The compact blocks stay compact in RAM, and dequantization happens inside the dot-product inner loop, one block at a time, into registers that are discarded immediately.
The kernels go further: to multiply a quantized weight row by an activation vector, the activations are first quantized on the fly to a companion 8-bit block format (Q8_K), and the dot product runs in integer SIMD arithmetic directly between the two block layouts, applying the per-block scales only at the end. Why bother? CPU decode is memory-bandwidth-bound — the machine spends its time hauling weights from RAM, not doing math. Reading 4.5-bit blocks instead of 16-bit floats cuts that traffic by more than 3×, which is why a quantized GGUF is not just smaller but faster on a CPU: the format and the kernel are designed together.