The residual stream as a communication channel
Every layer of a transformer reads from and writes to one shared object: the residual stream, a tensor x: [n_ctx, d_model] that carries one d_model-dimensional vector per token position. Crucially, sublayers add to it — x → x + attn(x), then x → x + mlp(x) — rather than overwriting it. Because the updates are purely additive, the stream behaves like a shared bus or communication channel: any earlier component can deposit a signal, and any later component can pick it up, with no forced interaction in between.
This additivity is what makes circuits tractable. The output logits are a sum of contributions — embedding, every attention head, every MLP — so you can ask what a single component wrote and follow its signal downstream. The stream has finite width, though: d_model dimensions are a bandwidth budget that many heads must share, which is exactly why features end up superposed and why components learn to communicate in near-orthogonal subspaces.
Reading and writing: a head as a low-rank map
An attention head touches the stream through four learned matrices. It reads a low-dimensional view with W_Q, W_K, W_V (each projecting d_model → d_head, with d_head < d_model) and writes back with W_O (d_head → d_model). The value pathway, end to end, applies W_V then W_O, so the map from stream-in to stream-out has rank at most d_head.
Written per position, the head’s output is out = A · (x W_V) W_O, where A: [n_ctx, n_ctx] is the attention pattern. Two conceptually separate things are happening here: A decides from which positions information flows, and W_V W_O decides what content flows once a position is selected. Because both projections are low rank, each head reads from and writes to only a small subspace of the stream — the geometric basis for heads specializing and not colliding.
Factoring a head into two circuits
The central move of circuit analysis is to collapse those four matrices into two combined operators. The attention scores are scores = (x W_Q)(x W_K)^T = x (W_Q W_K^T) x^T, which motivates defining the QK circuit W_QK = W_Q W_K^T, a d_model × d_model matrix of rank d_head. Likewise the content pathway defines the OV circuit W_OV = W_V W_O, also d_model × d_model and rank d_head.
The whole head then reads compactly as out = softmax( x W_QK x^T / √d_head ) · x W_OV. This is the factorization that turns a head into something you can interpret: two bilinear forms on the residual stream, one governing the attention pattern and one governing the moved content. A key payoff of this split is that the two circuits are independent: they share no parameters, so ‘where a head looks’ and ‘what it does when it looks’ can be characterized separately and then composed into one story.
The QK circuit: where to attend
The QK circuit is a bilinear form that scores every (query position, key position) pair: score(i, j) = x_i W_QK x_j^T. Read left to right, x_i is the token doing the looking and x_j is the token being considered; the softmax over j (with causal masking and the 1/√d_head scale) turns scores into the attention weights.
Because W_QK is generally not symmetric, attention is directional: ‘A attends to B’ is a different statement from ‘B attends to A.’ To see what a head keys on, sandwich the circuit in the token basis: W_E W_QK W_E^T is a vocab × vocab matrix whose (a, b) entry is how strongly a token a wants to attend to a token b, ignoring positional and contextual effects. A head that attends to the previous occurrence of the current token, for instance, shows a strong diagonal here — the QK circuit is literally a lookup rule.
The OV circuit: what to move
Once the pattern decides where attention lands, the OV circuit decides what gets copied. For a token attended to with weight one, the head adds x_j W_OV to the destination’s residual stream. So W_OV is the map ‘given that I read position j, this is the vector I deposit downstream.’
The clean way to interpret it is again in the token basis, but now bridging input tokens to output logits: W_U W_OV W_E is a vocab × vocab matrix whose (a, b) entry is how much attending to token b pushes up the logit for predicting token a (W_U is the unembedding). A pure copying head — one that says ‘predict whatever I’m looking at’ — has a dominant positive diagonal in this matrix. The QK circuit found the token; the OV circuit promotes it. The two answer genuinely different questions.
A worked example of the shapes
Concrete numbers keep the algebra honest. Take a small model with d_model = 512, n_heads = 8, and therefore d_head = 64. Then W_Q, W_K, W_V are each 512 × 64 and W_O is 64 × 512.
The QK circuit W_QK = W_Q W_K^T is 512 × 512 but its rank is at most 64: it is a full-width operator squeezed through a 64-dimensional bottleneck. Same for W_OV. So although each combined circuit is nominally a quarter-million-entry matrix, it carries only about 2 × 512 × 64 ≈ 65{,}000 free parameters — the same count as the underlying factors. The low rank is not a defect; it is a hard constraint that forces each head to read and write in a 64-dimensional slice of the 512-wide stream — which is precisely why distinct heads coexist on the same bus without overwriting one another.
Composition: how heads talk across layers
Single heads are only the alphabet. Interesting behavior comes from composition: a head in a later layer reading a signal that an earlier head wrote into the stream. Because the stream is additive, an early head’s OV output becomes part of the input to a later head’s Q, K, or V projection — giving three composition types.
Q-composition: the later head’s query is shaped by an earlier write, changing what it searches for. K-composition: the earlier write changes a token’s key, changing whether other tokens attend to it. V-composition: the earlier write changes what content the later head moves. Algebraically these are just the earlier W_OV appearing inside the later circuit — e.g. K-composition contributes a term like W_Q (W_OV^{early})^T W_K^T to the effective QK form. Composition is what lets two rank-64 heads implement a rule neither could express alone — effectively forming a virtual head whose circuits are products of the participants’. It is the formal reason a two-layer model is qualitatively more capable than a one-layer one: only at two layers does K-composition, and therefore induction, become possible.