Grokking is one of the strangest curves in deep learning. You train a small network on a simple algorithmic task; within a few hundred steps it fits the training set perfectly and its validation accuracy sits at chance. Classic intuition says stop — the model has overfit and nothing more will happen. But if you keep training, tens of thousands of steps later, long after the training loss flatlined, validation accuracy suddenly leaps from near-random to near-perfect. The model appears to abruptly ‘get it.’ Nina Power and colleagues at OpenAI named this grokking in their 2021 paper, and it has become a favourite laboratory for studying how — and when — generalization actually happens. This piece is the ground-level tour: what the phenomenon is, the experiment that surfaced it, the shape of the telltale curve, and the training conditions that make it appear. The companion articles go deeper into the internal mechanism and the formal theory; here we build the picture you need before either makes sense.
What grokking is
Grokking is delayed generalization: a model reaches perfect training accuracy early, plateaus at poor validation accuracy for a long time, and then — with no change to the data, the loss target, or the optimizer — generalizes suddenly and dramatically far into training. The word (borrowed from Heinlein’s Stranger in a Strange Land, where to grok is to understand something completely) captures the feeling of a network that memorizes first and comprehends later.
The defining feature is the gap in time between fitting the training set and generalizing to held-out data. In ordinary training these two events are roughly simultaneous: validation accuracy climbs alongside training accuracy. In grokking they are separated by orders of magnitude — training accuracy hits 100% at step ~10^3, while validation accuracy stays flat until step ~10^5. That separation is what makes the phenomenon worth a name, and a clean testbed for one question: when does a model stop memorizing and start understanding?
The experiment that surfaced it
Power et al. trained small transformers on algorithmic tasks over small finite groups — most famously modular arithmetic. A typical task: given tokens for a, an operator, and b, predict (a ∘ b) mod p for a prime like p = 97. Every input is a short sequence such as a ∘ b = ?, and the model must output the single correct residue in {0, 1, …, 96}.
These tasks are deliberately tiny and fully specified. There are only 97 × 97 = 9409 possible equations for binary mod-97 addition, so the entire ‘universe’ of the problem fits in memory. That is the point: because the task has a clean underlying rule and a small closed dataset, you can split it into a training fraction and a held-out fraction, train to convergence, and watch precisely whether the network has learned the rule or merely the table. Modular arithmetic, permutation composition, and similar group operations all show the effect, which is why they became the standard grokking benchmarks.
The signature curve
Plot training and validation accuracy against training steps on a log x-axis and you get grokking’s fingerprint. Two curves, three acts:
train acc ____----======================== (100% by ~10^3 steps)
/
val acc ____/______________________---- (chance until ~10^5, then leaps)
| | |
fit long plateau sudden
train (overfit-looking) generalizationIn act one the training accuracy shoots to 100% while validation sits at chance — the model has memorized the training equations. In act two, the long plateau, nothing visible improves on the validation side even as training continues for tens of thousands of steps; a practitioner watching only validation would conclude the run is dead. In act three validation accuracy rises steeply, often to near 100%, over a relatively short window. The transition is sharp enough that on a linear x-axis it looks almost like a step function.
Why it defies the usual intuition
The reason grokking is startling is that it violates the mental model most of us carry about overfitting. The standard story: once training accuracy is perfect and validation accuracy is poor and no longer improving, the model has overfit, further training only memorizes harder, and the right move is early stopping. Grokking is a flat contradiction — the model sits in exactly that ‘overfit and stuck’ state and then generalizes anyway.
It also complicates the tidy version of the bias–variance and double-descent stories. Here generalization is not gated by adding parameters or data; the architecture and dataset are fixed throughout. The only thing that changes is time. That tells us something important: for these tasks a memorizing solution and a generalizing solution both drive training loss to zero, so training accuracy cannot distinguish them. The optimizer initially settles into the memorizing solution and only later drifts toward the generalizing one — which means understanding grokking is really about understanding what keeps pushing the weights after the training loss is already gone.
The role of weight decay
The single most important knob for producing grokking is weight decay (L2 regularization). In the original experiments, runs with appropriate weight decay grokked reliably and comparatively early; runs with little or no weight decay often did not generalize within any practical budget at all — they stayed memorized essentially forever.
The intuition, at the overview level, is that once training loss is near zero the data term stops exerting force on the weights, but the weight-decay term keeps pulling them toward smaller norm. Among all the parameter settings that fit the training data perfectly, weight decay biases the optimizer toward the lowest-norm ones — and for these algorithmic tasks the low-norm solutions happen to be the ones that implement the general rule rather than a lookup table. Grokking, in this framing, is the slow migration from a high-norm memorizing solution to a low-norm generalizing one, driven by regularization long after the loss curve looks finished. The exact dynamics are the subject of the companion articles; the headline is that without a norm-shrinking pressure, the sudden generalization typically does not arrive.
Dataset fraction and the critical threshold
The second decisive factor is how much of the task you show the model. Power et al. swept the training-set fraction — the share of all possible equations placed in the training split — and found a striking pattern. Below a critical fraction the model never generalizes; above it, it eventually groks; and near the threshold the time-to-grok balloons.
Concretely, with too few examples there is simply not enough constraint to pin down the underlying rule — many different tables are consistent with the training data, so no amount of training recovers the true operation. As you add data past the critical point, the rule becomes the most economical explanation and grokking kicks in, sooner the more data you provide. The relationship is sharp rather than gradual: a few percentage points of extra training fraction can move a run from ‘never generalizes’ to ‘groks in a fraction of the steps.’ This data-efficiency threshold is one of grokking’s most robust and reproducible features.
A worked example: mod-97 addition
Make it concrete. Take addition modulo the prime p = 97. The full task is every pair (a, b) with 0 ≤ a, b < 97, labelled by (a + b) mod 97 — 9409 equations in total. Suppose you put 50% in the training split (~4700 equations) and hold out the rest.
Early in training the network memorizes those 4700 answers: training accuracy 100%, validation accuracy about 1/97 ≈ 1% — chance. A run watched only on validation looks like a failure for a very long time. Then, tens of thousands of steps later, validation accuracy climbs toward 100%: the model now answers (a + b) mod 97 correctly for pairs it was never shown, including combinations of a and b that never co-occurred in training. It could only do that by having internalized the operation — the general rule for modular addition — rather than the 4700 memorized rows. That behavioural jump, from table to rule, is grokking measured at the output.
Not just a toy: where else it shows up
Grokking was discovered on small algorithmic tasks, and it is fair to ask whether it is a curiosity of that setting. The honest answer is that the clean, dramatic version — a flat plateau followed by a near-vertical jump — is most reliably seen on these small, fully-specified problems, where a single crisp rule underlies a small dataset and you can train far past convergence cheaply.
That said, the underlying tension — a model that fits its training data long before it generalizes, with generalization improving well after training loss looks done — is not unique to toy tasks, and grokking is now widely used as a controlled microcosm for studying it. The value of the modular-arithmetic experiments is not that real training looks exactly like them, but that they isolate a real effect cleanly enough to study — the same reason physicists like frictionless planes.
Practical implications for training and monitoring
Even if you never train a mod-97 classifier, grokking carries a working lesson: a flat validation curve is not proof that a run is finished. The reflex to early-stop the moment validation stalls is usually correct on large, noisy datasets, but grokking is the standing counterexample — the very shape that signals ‘dead run’ can precede a sudden win on tasks with clean latent structure and strong regularization.
For CPU-scale and small-model work, three habits follow. First, log on a log-scale time axis and keep training past apparent convergence when a task is small and rule-like — the transition can hide many decades out. Second, treat weight decay as a first-class hyperparameter; it is often the difference between generalizing and never generalizing. Third, remember that training accuracy is a weak signal once it saturates — two models at 100% training accuracy can be worlds apart, and only held-out performance tells them apart.
Common misconceptions
A few pitfalls trip up newcomers. ‘Grokking means just train longer.’ Not by itself — without adequate weight decay and enough of the dataset, longer training below the critical fraction generalizes never, not eventually. Time is necessary, not sufficient.
‘It is only overfitting that eventually corrects.’ The opposite: overfitting normally worsens generalization with more training, whereas grokking improves it long after the fit is perfect, which is precisely why it is surprising. ‘It happens in every model.’ The clean effect is a property of specific task and regularization regimes, not a universal law. Finally, ‘the sudden jump means the model was idle during the plateau.’ It was doing plenty — the internal reorganization is gradual and hidden; only the behavioural readout is abrupt.
(a + b) mod p, a network memorizes the training set early — 100% train accuracy, chance-level validation — sits on that plateau for orders of magnitude more steps, and then suddenly generalizes to near-perfect held-out accuracy. The two ingredients that make it appear are weight decay, which keeps pushing the weights toward a low-norm, rule-implementing solution after the training loss is gone, and enough of the dataset to cross a sharp critical fraction below which generalization never comes. The practical lesson travels beyond the toy: a flat validation curve is not proof a run is finished, training accuracy is a weak signal once it saturates, and weight decay deserves to be tuned as a first-class knob. What the network rewires internally during the jump, and the formal account of why, are the subject of the companion pieces.