What the hypothesis actually claims

State it precisely, because the wording carries the whole idea. A randomly-initialized dense network f(x; θ_0) contains a sparse subnetwork — specified by a binary mask m over its weights — such that training only the masked weights m ⊙ θ_0 from that same initialization reaches test accuracy comparable to the fully trained dense network, in a comparable number of iterations, while using a small fraction of the parameters.

Two words do the heavy lifting. Same: the subnetwork keeps the exact weight values it was born with, θ_0, not fresh random values. Trainable: it is not merely a good network after the full model has done the learning — it can be trained in isolation to the same quality. That combination is what makes LTH a statement about initialization, not just about compression. The dense network is like a bundle of lottery tickets; training reveals which ticket held the winning numbers from the start.

Advertisement

Winning tickets, defined

A winning ticket is the pair (m, θ_0): a mask that selects which connections survive, together with the specific initial values those connections started with. Both halves matter. Keep the mask but randomly re-initialize the surviving weights and the magic disappears — the subnetwork trains slower and to lower accuracy. Keep the init but change the mask to a random one of the same size and it also fails. The winning combination is a particular sparse structure paired with a particular set of starting values.

How sparse? In the original experiments on small vision networks, winning tickets routinely retained only 10–20% of the weights — sometimes as little as 3.6% — while matching or briefly exceeding the dense model’s accuracy. The remaining 80–90% of connections were, in hindsight, dead weight that the network never needed. LTH says that dead weight was identifiable in principle, and that a lean sub-model with the right init was waiting inside the whole time.

Advertisement

Finding a ticket: magnitude pruning

How do you locate the winning subnetwork? Frankle and Carbin use the simplest reasonable heuristic: magnitude pruning. Train the dense network to completion, then remove the connections whose final weights are smallest in absolute value — the intuition being that a weight that ended up near zero was contributing little. The one-shot recipe is four steps:

1. Initialize dense net with weights θ_0 (save this copy).
2. Train the dense net to convergence, obtaining θ_final.
3. Prune the p% of weights with smallest |θ_final| → mask m.
4. Reset surviving weights to their values in θ_0: m ⊙ θ_0.
   Train this sparse net — it is the candidate winning ticket.

Step 4 is the counter-intuitive one. You use the trained weights only to decide the mask — which connections to keep — and then you throw the trained values away and rewind the survivors to θ_0. The claim is that this rewound sparse net, retrained, matches the dense original.

Iterative magnitude pruning

One-shot pruning at high sparsity is crude: cut 80% of the weights in a single pass and you often overshoot, damaging the network. The stronger and more reliable procedure is iterative magnitude pruning (IMP), which reaches the same target sparsity gradually over several rounds.

repeat for n rounds:
    train the (currently sparse) net to convergence
    prune a small fraction (e.g. 20%) of the smallest
        surviving weights, extending the mask m
    reset all surviving weights to θ_0
# after n rounds, sparsity ≈ (1 - 0.8^n)

Each round prunes only a little, so the network re-adapts before the next cut. Removing 20% per round, after 5 rounds you keep 0.8^5 ≈ 33% of weights; after 10 rounds, about 11%. IMP consistently finds tickets at far higher sparsity than one-shot pruning can — the gentle, repeated squeeze is what lets the tiny 10–20% subnetworks emerge without collapsing. The cost is real: you retrain the network once per round, so producing a ticket is more expensive than training the dense model once.

The control experiment that gives it teeth

A claim like this needs a control, or it is just "pruning works." The decisive experiment is random reinitialization. Take the exact same winning mask m, but instead of resetting the survivors to their original θ_0, draw fresh random values from the same distribution. Train that. If the mask alone were responsible, this random-init version would train just as well.

It does not. The randomly re-initialized subnetwork learns more slowly and plateaus at noticeably lower accuracy, and the gap widens as sparsity increases. That single result is the backbone of the whole hypothesis: it isolates the initialization as an active ingredient, not an afterthought. The pruned structure is necessary but not sufficient — it must be married to the particular starting weights the survivors were dealt.

A worked sense of the numbers

Put concrete figures on it. Suppose a small fully-connected network for MNIST has W = 266K weights and trains to 98.0% test accuracy in, say, 20K iterations. Run IMP at 20% per round for 10 rounds. Sparsity climbs 0 → 20% → 36% → … → ~89%, so the final ticket keeps roughly 0.8^10 × 266K ≈ 29K weights.

The reported behavior: that ~11%-density ticket, trained from θ_0, still reaches about 98% — often in fewer iterations than the dense net, and sometimes at slightly higher accuracy at moderate sparsities before declining once you prune too aggressively. Meanwhile the random-reinit control at the same density lands a point or more lower and trains slower. That curve — match or beat dense down to some sparsity, then fall off — is the empirical signature of a winning ticket.

Why this was a genuine surprise

Compression was old news. You could already prune a trained network to a fraction of its size and, with fine-tuning, keep most of its accuracy. The received wisdom, though, was that you needed the big network to learn: the over-parameterization was thought to be essential during optimization, even if the end result was compressible. Train a small network from scratch and it would underperform.

LTH punctures that. It shows a sparse network that trains from scratch to full accuracy — the catch is only that you must start it from the right init, and finding that init currently requires having trained the big network first. So over-parameterization still buys you something: not raw capacity, but a larger pool of candidate tickets, raising the odds that some sparse subnetwork got a lucky, trainable initialization. Big networks may work partly because they contain many lottery tickets and only need one to win.

Why it matters

The significance runs in three directions. Scientifically, LTH reframes over-parameterization and gives a concrete, testable object — the winning ticket — for studying why big networks optimize well, turning a vague intuition ("bigger is easier to train") into experiments with controls.

For sparsity research, it energized the search for sparse networks that are cheap to train, not just cheap to deploy. Classic pruning shrinks a model after the expensive training is done; LTH raised the tantalizing question of whether you could identify a trainable sparse network early and skip most of the dense compute. Practically, it sharpened the notion that most of a network’s weights are, in the end, unnecessary — a premise that underlies modern interest in sparse and small models.