The information bottleneck (IB) method, introduced by Naftali Tishby, Fernando Pereira and William Bialek in 1999, is a precise answer to a vague-sounding question: given a signal X and something you care about, Y, what is the most compressed summary of X that still tells you as much as possible about Y? It reframes lossy compression around relevance rather than reconstruction, and packs the whole trade-off into one Lagrangian with a single knob, β. This article stays with the method itself — the objective, the self-consistent equations that solve it, the information curve it traces, and its kinship with rate-distortion theory — rather than the later, much-debated story about what happens inside a training deep network, which is covered in the companion piece on bottleneck theory.
The question: what is relevant information
Classical lossy compression asks how few bits can represent X so that X can be reconstructed within some tolerance. But often we do not care about reconstructing X at all — we care about some other variable Y that X is informative about: the word a sound encodes, the topic of a document, the label of an image. The trouble with ‘keep the relevant part’ is that relevant is undefined until you say relevant to what.
The IB method makes it concrete by naming that second variable. Relevance is measured by the mutual information I(T; Y) between a compressed representation T and the target Y; cost is measured by I(X; T), the bits of X the representation spends. Given only the joint p(x, y), the method asks for the mapping that buys the most relevance per bit of complexity — no distance function on X required, since the target supplies the meaning.
Two mutual informations
Everything is scored by two quantities. Mutual information between variables A and B is
I(A; B) = Σ_a Σ_b p(a,b) log( p(a,b) / (p(a) p(b)) )
= H(A) − H(A | B)measured in bits (log base 2) or nats. It is zero exactly when A and B are independent and grows as one becomes more predictable from the other. In IB, I(X; T) is the rate or complexity of the representation while I(T; Y) is the relevance it preserves about the target. The two pull against each other: the cheapest representation (constant T) scores zero on both, and the richest (T = X) maximises relevance but pays the full complexity I(X; T) = H(X). The interesting maps live between those extremes.
The IB Lagrangian
The method encodes the trade-off as a single functional to minimize over the stochastic map p(t | x):
L[p(t|x)] = I(X; T) − β · I(T; Y)The first term is the number of bits you would like to squeeze out; the second is the relevant information you would like to keep. The Lagrange multiplier β ≥ 0 is the exchange rate between them. Crucially the optimisation is over soft assignments p(t | x) — a probability of mapping each input to each code symbol — not a hard clustering; that soft formulation is what makes the objective smooth and gives it a clean stationary-point solution. Minimising L at fixed β yields one optimal representation; sweeping β yields a whole family, one per operating point on the trade-off.
Beta: the compression versus prediction knob
β is the single dial that tunes the whole method. At β → 0 the objective is dominated by the I(X; T) term, so the cheapest solution wins outright: collapse everything to one symbol, I(X; T) = I(T; Y) = 0. Maximum compression, zero prediction. As β grows, retaining relevant information becomes worth paying bits for, and T is allowed to make finer distinctions among inputs that differ in what they say about Y.
At β → ∞ the relevance term dominates and the method preserves all the predictive information it can, driving T toward a minimal sufficient statistic of X for Y — the smallest representation with I(T; Y) = I(X; Y). Reading β as an inverse temperature is exact: small β is ‘hot’ and blurs everything together, large β is ‘cold’ and freezes sharp distinctions.
The Markov constraint
The representation is only allowed to see the input, not the target directly. Formally T is defined by a channel from X alone, so the three variables form a Markov chain:
T ← X ← Y (equivalently Y → X → T)This says T is conditionally independent of Y given X: whatever T learns about the target, it must learn through X. The immediate consequence is the data processing inequality, I(T; Y) ≤ I(X; Y) — no compression can manufacture relevance absent from the input. That ceiling I(X; Y) is the most any bottleneck representation could predict, and the IB solution’s job is to approach it as cheaply as possible in I(X; T).
The self-consistent equations
Taking the variational derivative of L with respect to p(t | x), subject to normalisation and the Markov constraint, and setting it to zero gives not a closed form but three coupled equations that an optimum must satisfy simultaneously:
p(t|x) = p(t) / Z(x, β) · exp( −β · D_KL[ p(y|x) || p(y|t) ] )
p(t) = Σ_x p(x) p(t|x)
p(y|t) = Σ_x p(y|x) p(x|t)Here Z(x, β) is a per-input partition function that normalises the assignment, and D_KL is the Kullback-Leibler divergence. The first equation is the heart of it: an input x is assigned to code t with weight that decays exponentially in how badly t’s predictive profile p(y|t) disagrees with x’s own p(y|x). Inputs that imply the same thing about Y are drawn to the same code.
An emergent distortion measure
Notice what plays the role of ‘distance’ in that assignment rule: the KL divergence d(x, t) = D_KL[ p(y|x) || p(y|t) ]. This is the single most elegant feature of the method. In ordinary clustering you must choose a distortion measure — Euclidean, cosine, whatever — and the answer depends heavily on that arbitrary choice. IB does not ask you to pick one. The relevant distortion emerges from the data: two inputs are close precisely when they carry the same distribution over the target.
So the effective geometry on X is induced entirely by p(y | x). A representation groups inputs that are predictively interchangeable, even if they look nothing alike in raw feature space — clustering by relevance, with a similarity metric discovered rather than assumed. That is exactly what ‘relevant’ needed all along.
Solving by iteration
Because the three equations are mutually dependent — p(t|x) depends on p(y|t), which depends on p(x|t), which depends back on p(t|x) — they are solved by fixed-point iteration, a generalisation of the Blahut-Arimoto algorithm from rate-distortion theory. Fix β, initialise the assignments, then repeat:
repeat until convergence:
update p(t|x) from current p(t), p(y|t) # assignment step
update p(t) = Σ_x p(x) p(t|x) # marginal
update p(y|t) = Σ_x p(y|x) p(x|t) # centroid stepEach full sweep is guaranteed not to increase the functional L, so the iteration converges to a locally optimal self-consistent solution. It alternates like an EM or k-means loop: the assignment step is the expectation-like move, and the p(y|t) update recomputes each code’s ‘centroid’ — here a distribution over Y, not a point in space. Because L is non-convex in the joint variables, restarts or annealing help avoid poor optima.
The information curve
Solve the method across all β and plot the achieved I(T; Y) against the spent I(X; T). The upper boundary of the achievable region is the information curve (also called the relevance-compression curve), and it is the true object the method characterises. It is concave and monotonically non-decreasing: more complexity can only buy more — or equal — relevance, with diminishing returns.
Two features anchor it. It starts at the origin (zero bits, zero relevance) and saturates at (H(X), I(X; Y)), beyond which extra complexity is wasted because no relevant information remains to extract. And its slope at any point equals 1/β, so β literally selects the operating point: a steep stretch (small β) is where each extra bit buys a lot of relevance; the flattening tail (large β) is where bits are nearly spent.
Kinship with rate-distortion theory
IB is best understood as a relative of Shannon’s rate-distortion theory, with one decisive twist. Rate-distortion minimises the rate I(X; T) subject to an average distortion constraint E[d(x, t)] ≤ D, where d is a distortion function you specify in advance. The IB Lagrangian has exactly the same shape — minimise rate, penalised by a quality term — and it is solved by the same Blahut-Arimoto machinery.
The twist is that IB does not accept a hand-picked d. It fixes the quality side as ‘preserve information about Y’ and lets the distortion D_KL[p(y|x) || p(y|t)] fall out of that requirement. In effect, rate-distortion compresses X to reconstruct X; IB compresses X to reconstruct the conditional p(y | x). Swapping a chosen fidelity criterion for a learned, task-defined one is the whole conceptual contribution.
A tiny worked example
Let X be uniform over four documents and Y a binary topic. Suppose two documents are strongly topic-A (p(y=A|x) ≈ 0.9) and two are strongly topic-B (p(y=A|x) ≈ 0.1). With four codes and large β, the identity map keeps I(X; T) = 2 bits and full relevance. Now lower β.
The KL assignment rule notices that the two topic-A documents have nearly identical p(y|x), so D_KL between them and a shared code is tiny — merging them costs almost no relevance while halving the code count. The iteration collapses each topic pair into one code, landing at I(X; T) = 1 bit with I(T; Y) essentially unchanged. The method spends its bit budget separating predictively different inputs and refuses to spend any distinguishing inputs that say the same thing about the target.
Why the method endures
The IB method matters because it gives a first-principles, assumption-light definition of a ‘good’ representation: the one on the information curve for your chosen β. It needs only the joint p(x, y), invents its own relevance metric, and reduces the fuzzy goal of ‘keep what matters’ to a solvable optimisation. The vocabulary it introduced — complexity versus relevance, minimal sufficient statistics, an information curve with slope 1/β — is now standard.
Its limits are honest ones. The exact solution needs p(x, y), which for high-dimensional or continuous data must be estimated — the motivation for later variational and neural relaxations — and the iteration finds local, not global, optima. But as a clean statement of what compression for a purpose means, the 1999 formulation remains the reference point everything else is measured against.