Grouped-query attention (GQA) gives you a single dial — the number of key/value groups g — that trades a sliver of model quality for a large, permanent cut in serving cost. Its sibling piece derives why the KV cache shrinks by a factor of h/g. This one is the engineering companion: given that GQA works, how do you actually choose g? The honest answer weaves together four things that rarely get discussed at once — the roofline reason fewer K/V heads make decode faster, how many concurrent requests a fixed memory budget then buys you, how g must line up with your tensor-parallel layout, and why you can bolt GQA onto an already-trained model instead of pretraining from scratch. We finish by placing GQA next to its neighbours, MQA and multi-head latent attention, so you know when a different lever is the better one.
One dial from MHA to MQA
The whole family is a single continuum. Multi-head attention (MHA) gives every one of the h query heads its own key and value head, so the cache stores h K/V pairs per token per layer. Multi-query attention (MQA) collapses that to a single shared K/V pair for all heads. GQA sits between them: keep all h query heads, but partition them into g groups that each share one K/V pair.
So g = h is MHA, g = 1 is MQA, and any divisor of h in between is a valid GQA design. Crucially the query side never changes — queries are recomputed each step and thrown away, so they are cheap. The only thing g moves is how many distinct K and V vectors accumulate in the cache. That single fact is why choosing g is almost entirely a systems decision about memory and bandwidth, with only a light touch of modelling quality on the other side of the scale.
Why fewer K, V heads is the right lever
Why attack the number of K/V heads specifically, and not, say, the head dimension? Because autoregressive decode is memory-bandwidth-bound, and the KV cache is the part of the byte traffic that grows without bound. Each decode step must stream two things out of memory: the model weights (once) and the entire KV cache (once). The arithmetic per token is tiny by comparison, so time-per-token tracks bytes moved, not FLOPs.
The useful way to see this is arithmetic intensity — FLOPs performed per byte read. Roofline analysis says throughput is bandwidth-limited whenever intensity sits below the hardware’s FLOP-to-bandwidth ratio, which decode almost always does. Shrinking the KV cache by h/g cuts the bytes each step reads, which raises arithmetic intensity and pushes decode toward the compute-bound regime where the accelerator is actually busy. Reducing d_head instead would shrink model capacity everywhere; reducing K/V heads targets exactly the bandwidth bottleneck and leaves the query-side expressivity intact. That surgical fit is why GQA, not some other trim, became the standard.
What the smaller cache buys: a throughput example
The cache saving translates directly into concurrency, which is where serving economics live. Take a 70B-class model: L = 80 layers, d_head = 128, bf16 (p = 2 bytes). Per token per layer a single K/V head costs 2 · d_head · p = 512 bytes, so one sequence at S = 8192 tokens holds:
per-seq KV = 2 · L · n_kv · d_head · S · p
MHA (n_kv = 64): 2·80·64·128·8192·2 ≈ 20.0 GiB / sequence
GQA (n_kv = 8): 2·80· 8·128·8192·2 ≈ 2.5 GiB / sequenceNow suppose that after loading weights you have a 40 GiB budget left for KV cache on the accelerator. Under MHA you fit just 40 / 20 = 2 concurrent 8k-token sequences; under GQA with g = 8 you fit 40 / 2.5 = 16. That is an eightfold jump in batch size from the same memory — and because decode throughput scales with how many sequences you can run in parallel before hitting the memory wall, it is roughly an eightfold jump in tokens-per-second of serving capacity too. The cache dial is really a throughput dial.
The same arithmetic explains long context from the other direction: if you fix the batch size instead of the memory, GQA lets each sequence carry roughly h/g times more tokens of history within the budget. So one choice of g simultaneously widens how many users you serve and how much context each of them gets — you spend the saving on whichever axis your product needs more.
Choosing g on the quality-versus-cache Pareto
If bigger g means less sharing and more quality, and smaller g means more saving, the sweet spot is wherever the two curves cross usefully. The empirical finding from Ainslie et al. is that the trade is pleasantly asymmetric: moving from MHA down to a modest number of groups recovers almost all of MQA’s memory and speed win while giving up close to nothing in accuracy, whereas the last step down to g = 1 (full MQA) is where quality and training stability actually start to suffer.
In Pareto terms, g = 8 sits at the knee of the frontier — you have already banked most of the cache reduction, and paying more (smaller g) buys diminishing memory returns at rising quality cost. That is why eight has become the de-facto default rather than one. The rule of thumb: pick the smallest g that keeps evaluation metrics within noise of your MHA baseline, and do not chase the last factor of two in cache size — it is the expensive half of the curve.
How g interacts with tensor parallelism
There is a second, systems-level constraint on g that is easy to miss until deployment. Large models are served with tensor parallelism (TP): the attention heads are sharded across several GPUs. Query heads divide cleanly, but the n_kv = g K/V heads have to be distributed too, and the clean case is one (or an integer number of) K/V heads per GPU.
When g is at least the TP degree and divisible by it, each GPU owns whole K/V heads and no coordination is needed. When g is smaller than the TP degree — say g = 1 (MQA) on 8 GPUs — the shared K/V must be replicated across ranks, which costs a little memory and complicates the kernel. A group count of eight is convenient precisely because it maps one K/V head per GPU on the common 8-way TP setup. This is a happy alignment, not the reason eight was chosen — the quality/cache knee came first — but it does mean you should sanity-check g against your intended TP layout before committing to a number.
Uptraining: adding GQA without starting over
You do not have to pretrain a GQA model from scratch to get these benefits. The cheap path, and the one most teams take, is to convert an existing multi-head checkpoint: for each group, mean-pool the h/g original key projections into one shared key projection, do the same for values, and leave the query projections untouched. Averaging preserves the behaviour the model already learned far better than dropping heads or reinitialising.
After pooling, the model is close but not quite right, so you uptrain — continue pretraining on a small slice of the original data budget, on the order of a few percent — and quality snaps back to near-MHA. The economic consequence for choosing g is that experimentation is cheap: you can pool a trained MHA model to several candidate group counts, briefly uptrain each, and read the quality/cache trade off directly on your own evaluations rather than trusting a paper’s defaults. The sibling article works this conversion through in more detail.
GQA vs MQA vs MLA
GQA is not the only way to shrink the KV cache, and knowing the neighbours keeps you from over-fitting to one lever. MQA is simply GQA at g = 1: maximum saving, but the shared single K/V head is a real quality and stability risk, so it survives mostly in latency-critical or smaller-model settings where the cut is worth it.
Multi-head latent attention (MLA), introduced with DeepSeek-V2, takes a different route entirely: instead of reducing the number of K/V heads, it caches a single low-rank latent vector per token and up-projects it back into full keys and values on the fly. That can reach a smaller cache than GQA at comparable quality, at the price of extra projection compute and a more intricate implementation. The mental model: GQA shares K/V across heads, MLA compresses K/V into a latent and reconstructs it. GQA wins on simplicity and broad tooling support and is the safe default; MLA is worth reaching for when the KV cache is still your binding constraint after GQA. In practice most open-weight models — the Llama, Mistral, and Qwen families among them — land on GQA precisely because it is a small, well-understood change to a standard attention block that any inference stack already supports, and the quality cost is close enough to zero that it is rarely the thing worth optimising further.
Practical notes and pitfalls
A few things to keep straight when you commit to a value of g. Divisibility: g must divide h evenly, and for painless serving it should also line up with your TP degree. Broadcast vs. fused kernel: a naive implementation repeats each group’s K/V h/g times to rebuild MHA-shaped tensors — correct, but it throws away the bandwidth win inside the attention kernel; a fused GQA kernel reads the shared K/V once, which is where the real decode speedup lives. Prefill is unchanged: GQA is a decode-memory optimisation, so do not expect it to cut the compute-bound prefill pass meaningfully.
For small models on CPU the calculus tilts even further toward GQA: RAM bandwidth is scarcer than on a GPU, so an h/g-times smaller cache both fits more comfortably in fast cache levels and moves fewer bytes per generated token — the phase that dominates the interactive latency a user actually feels. Choose g once, with the roofline and your deployment in view, and it pays off on every token thereafter.