From tensors on disk to a running model
A trained model is, concretely, a few hundred named tensors plus the metadata needed to run them: architecture, layer and head counts, the RoPE and normalization settings, and the entire tokenizer. Earlier formats scattered this across a pickle file, a JSON config, a tokenizer blob, and a separate weights file — fine for a Python training rig, awkward for a single C++ binary that wants to load and run in milliseconds.
GGUF (the successor to GGML and GGMF) collapses all of that into one file with a design goal that shapes everything else: the loader should memory-map the file and use tensors with no parsing and no copying. That constraint explains the layout — a typed key/value header, a tensor directory, then raw quantized tensor data laid out contiguously and aligned so the CPU reads it directly. GGUF is how that zero-copy promise is kept; k-quants are what those raw tensor bytes contain.
The container: a typed header, then tensors
A GGUF file has three parts. First a small header: the magic bytes GGUF, a version number (v3 is current), the tensor count, and the metadata key/value count. Second, the metadata — an ordered list of typed key/value pairs. Values are strongly typed (u32, f32, bool, string, nested arrays), so the reader never has to guess. Keys are namespaced strings like llama.attention.head_count, llama.block_count, and the tokenizer.ggml.* family carrying the full vocabulary and merges.
Third comes the tensor directory: for each tensor, its name, dimension count, shape, its ggml_type (the quantization format), and a byte offset into the data region — then the tensor data itself. Because every hyperparameter and the tokenizer travel inside the file, a GGUF is fully self-contained.
Why the layout is mmap-friendly
The offsets in the tensor directory point into a data region where each tensor is padded to an alignment boundary (32 bytes by default, recorded as general.alignment). That alignment is not cosmetic: it lets the quantization kernels issue aligned SIMD loads.
When llama.cpp opens the model it calls mmap. The OS maps the bytes into the process’s address space without reading them; pages are faulted in lazily as the compute touches them. Nothing is deserialized or copied, and the weights are never expanded to fp32 — the in-memory bytes are the on-disk bytes. So startup is near-instant even for a multi-gigabyte model, and several processes mapping the same file share one physical copy in the OS page cache. The compact quantized layout on disk is also the compact layout in RAM — exactly what a memory-bandwidth-bound CPU wants.
Block quantization, the base idea
Quantization stores weights in fewer bits than fp16, but you cannot just round each weight independently — you need a scale to map small integers back to real magnitudes. GGUF applies scales per block rather than per tensor, because a single scale for millions of weights would be swamped by the largest outlier. The legacy Q4_0 type is the clearest example: take 32 weights, find one fp16 scale d, and store each weight as a signed 4-bit integer q in [-8, 7], reconstructed as w = d · q.
The bookkeeping: 2 bytes for d plus 16 bytes of quants = 18 bytes for 32 weights, or 4.5 bits per weight. Add a per-block minimum (Q4_1: w = d · q + m) for an affine fit and it costs a little more. These flat, one-scale-per-block schemes are simple and fast but leave quality on the table. K-quants are the fix.
K-quants: the super-block and hierarchical scales
K-quants (introduced by Iwan Kawrakow in llama.cpp) keep block quantization but add a second level of scaling. A k-quant type groups QK_K = 256 weights into a super-block divided into sub-blocks of 32 or 16 weights. Each sub-block gets its own scale — and, for the affine types, its own minimum — so magnitude variation across the 256 weights is tracked far more finely than a single scale could.
The trick that makes this cheap is that the sub-block scales and mins are themselves quantized, typically to 6 bits, then rescaled by one fp16 super-block scale. So the hierarchy is: a top-level fp16 d (and dmin) for the whole super-block, low-bit sub-block scales/mins beneath it, and the low-bit weight quants at the bottom. You pay a few extra bits for that scale metadata but buy much better reconstruction — why k-quants beat legacy Q4_0-style types at equal size.
A worked Q4_K block layout
Q4_K is the workhorse. Its super-block holds 256 weights as eight sub-blocks of 32, and packs like this:
d : fp16 2 bytes super-block scale for the 8 scales
dmin : fp16 2 bytes super-block scale for the 8 mins
scales : 8 x 6-bit } 12 bytes packed: 16 six-bit values = 96 bits
mins : 8 x 6-bit } (scale_i and min_i per sub-block)
qs : 256 x 4-bit 128 bytes the weight quants (0..15)
----------------------------------------------------------------
total : 144 bytes -> 144 x 8 / 256 = 4.5 bits per weight
dequant, weight j in sub-block i:
w = d * scale_i * q_j - dmin * min_iRead that dequant line carefully: scale_i and min_i are the 6-bit values for the sub-block weight j lives in, and q_j is its 4-bit quant; the fp16 d and dmin lift those integers back to real magnitudes. The effective width, 4.5 bpw, is identical to Q4_0 — but the eight independent affine sub-block fits reconstruct the tensor far more faithfully. That is the whole k-quant bargain in one block.
Q5_K and Q6_K: spending the next bit
The other types vary the same super-block. Q5_K keeps the 256-weight layout and 6-bit sub-block scales/mins but stores 5-bit weight quants: the low 4 bits packed as in Q4_K (128 bytes) plus a 32-byte plane holding each weight’s 5th bit — 176 bytes per super-block, or 5.5 bpw, a clear quality bump for one extra bit.
Q6_K restructures slightly: 256 weights as sixteen sub-blocks of 16, each with a plain int8 scale. It stores the low 4 bits of every weight (128 bytes), the high 2 bits (64 bytes), sixteen int8 scales (16 bytes), and one fp16 d (2 bytes) — 210 bytes, or 6.5625 bpw. The same idea runs down to Q3_K and Q2_K, which lean even harder on the hierarchical scales to stay usable where a flat 2-bit quant would collapse.
Effective bits-per-weight, and a file-size example
Because each type carries fixed scale overhead per fixed-size block, its bits-per-weight is a constant you can compute exactly:
| Type | Block | Bytes/block | Effective bpw |
|---|---|---|---|
| Q2_K | 256 | 84 | 2.625 |
| Q3_K | 256 | 110 | 3.4375 |
| Q4_K | 256 | 144 | 4.5 |
| Q5_K | 256 | 176 | 5.5 |
| Q6_K | 256 | 210 | 6.5625 |
| Q8_0 | 32 | 34 | 8.5 |
Turn that into a real number. A 7-billion-parameter model in fp16 needs 7e9 × 16 / 8 ≈ 14 GB. At a uniform Q4_K (4.5 bpw) they are 7e9 × 4.5 / 8 ≈ 3.9 GB — a 3.5× shrink that turns a data-center model into a laptop model. Q6_K lands near 5.7 GB, Q2_K near 2.3 GB. This predictable scaling is why you can eyeball a GGUF filename and know both its footprint and roughly where it sits on the size-versus-quality curve.