What grokking is
Grokking is delayed generalization: a model reaches perfect training accuracy early, plateaus at poor validation accuracy for a long time, and then — with no change to the data, the loss target, or the optimizer — generalizes suddenly and dramatically far into training. The word (borrowed from Heinlein’s Stranger in a Strange Land, where to grok is to understand something completely) captures the feeling of a network that memorizes first and comprehends later.
The defining feature is the gap in time between fitting the training set and generalizing to held-out data. In ordinary training these two events are roughly simultaneous: validation accuracy climbs alongside training accuracy. In grokking they are separated by orders of magnitude — training accuracy hits 100% at step ~10^3, while validation accuracy stays flat until step ~10^5. That separation is what makes the phenomenon worth a name, and a clean testbed for one question: when does a model stop memorizing and start understanding?
The experiment that surfaced it
Power et al. trained small transformers on algorithmic tasks over small finite groups — most famously modular arithmetic. A typical task: given tokens for a, an operator, and b, predict (a ∘ b) mod p for a prime like p = 97. Every input is a short sequence such as a ∘ b = ?, and the model must output the single correct residue in {0, 1, …, 96}.
These tasks are deliberately tiny and fully specified. There are only 97 × 97 = 9409 possible equations for binary mod-97 addition, so the entire ‘universe’ of the problem fits in memory. That is the point: because the task has a clean underlying rule and a small closed dataset, you can split it into a training fraction and a held-out fraction, train to convergence, and watch precisely whether the network has learned the rule or merely the table. Modular arithmetic, permutation composition, and similar group operations all show the effect, which is why they became the standard grokking benchmarks.
The signature curve
Plot training and validation accuracy against training steps on a log x-axis and you get grokking’s fingerprint. Two curves, three acts:
train acc ____----======================== (100% by ~10^3 steps)
/
val acc ____/______________________---- (chance until ~10^5, then leaps)
| | |
fit long plateau sudden
train (overfit-looking) generalizationIn act one the training accuracy shoots to 100% while validation sits at chance — the model has memorized the training equations. In act two, the long plateau, nothing visible improves on the validation side even as training continues for tens of thousands of steps; a practitioner watching only validation would conclude the run is dead. In act three validation accuracy rises steeply, often to near 100%, over a relatively short window. The transition is sharp enough that on a linear x-axis it looks almost like a step function.
Why it defies the usual intuition
The reason grokking is startling is that it violates the mental model most of us carry about overfitting. The standard story: once training accuracy is perfect and validation accuracy is poor and no longer improving, the model has overfit, further training only memorizes harder, and the right move is early stopping. Grokking is a flat contradiction — the model sits in exactly that ‘overfit and stuck’ state and then generalizes anyway.
It also complicates the tidy version of the bias–variance and double-descent stories. Here generalization is not gated by adding parameters or data; the architecture and dataset are fixed throughout. The only thing that changes is time. That tells us something important: for these tasks a memorizing solution and a generalizing solution both drive training loss to zero, so training accuracy cannot distinguish them. The optimizer initially settles into the memorizing solution and only later drifts toward the generalizing one — which means understanding grokking is really about understanding what keeps pushing the weights after the training loss is already gone.
The role of weight decay
The single most important knob for producing grokking is weight decay (L2 regularization). In the original experiments, runs with appropriate weight decay grokked reliably and comparatively early; runs with little or no weight decay often did not generalize within any practical budget at all — they stayed memorized essentially forever.
The intuition, at the overview level, is that once training loss is near zero the data term stops exerting force on the weights, but the weight-decay term keeps pulling them toward smaller norm. Among all the parameter settings that fit the training data perfectly, weight decay biases the optimizer toward the lowest-norm ones — and for these algorithmic tasks the low-norm solutions happen to be the ones that implement the general rule rather than a lookup table. Grokking, in this framing, is the slow migration from a high-norm memorizing solution to a low-norm generalizing one, driven by regularization long after the loss curve looks finished. The exact dynamics are the subject of the companion articles; the headline is that without a norm-shrinking pressure, the sudden generalization typically does not arrive.
Dataset fraction and the critical threshold
The second decisive factor is how much of the task you show the model. Power et al. swept the training-set fraction — the share of all possible equations placed in the training split — and found a striking pattern. Below a critical fraction the model never generalizes; above it, it eventually groks; and near the threshold the time-to-grok balloons.
Concretely, with too few examples there is simply not enough constraint to pin down the underlying rule — many different tables are consistent with the training data, so no amount of training recovers the true operation. As you add data past the critical point, the rule becomes the most economical explanation and grokking kicks in, sooner the more data you provide. The relationship is sharp rather than gradual: a few percentage points of extra training fraction can move a run from ‘never generalizes’ to ‘groks in a fraction of the steps.’ This data-efficiency threshold is one of grokking’s most robust and reproducible features.
A worked example: mod-97 addition
Make it concrete. Take addition modulo the prime p = 97. The full task is every pair (a, b) with 0 ≤ a, b < 97, labelled by (a + b) mod 97 — 9409 equations in total. Suppose you put 50% in the training split (~4700 equations) and hold out the rest.
Early in training the network memorizes those 4700 answers: training accuracy 100%, validation accuracy about 1/97 ≈ 1% — chance. A run watched only on validation looks like a failure for a very long time. Then, tens of thousands of steps later, validation accuracy climbs toward 100%: the model now answers (a + b) mod 97 correctly for pairs it was never shown, including combinations of a and b that never co-occurred in training. It could only do that by having internalized the operation — the general rule for modular addition — rather than the 4700 memorized rows. That behavioural jump, from table to rule, is grokking measured at the output.