A training loss curve is the richest instrument you have while a model trains — a one-dimensional trace that quietly encodes your learning rate, your data quality, your batch size, and whether the run is healthy at all. The trouble is that most people glance at it, see the number going down, and move on, wasting almost all the signal. A loss curve has a shape, and the shape is legible: a healthy run traces a specific power-law descent, an unhealthy one leaves fingerprints — a warmup bump, a spike, a plateau, a widening gap between train and validation. This piece is about reading those shapes: what a good curve looks like, how the log-log view turns a boring dive into a straight diagnostic line, and how each deviation maps back to a cause. It is about interpretation — learning to look — rather than a catalogue of fixes.
The curve is an instrument, not a scoreboard
Treating the loss curve as a scoreboard — one number, lower is better, glance and go — wastes it. It is closer to an oscilloscope: every hyperparameter leaves a signature in the trace, and once you know the signatures you can diagnose a run from the shape alone. The vertical axis is the loss (for language models, mean cross-entropy per token over a batch); the horizontal axis is progress, usually optimizer steps or tokens seen.
Two traces matter. The training loss, on the batch just learned from, is noisy and immediate. The validation loss, on held-out data the model never trains on, is smooth and honest. Reading well means reading both: the fast, jittery line for stability, the slow, clean line for generalization.
What the number actually is
Reading the shape starts with knowing what the height means. The loss is cross-entropy in nats (natural log) or bits (log base 2): L = -(1/N) Σ_i ln p(x_i), the average negative log-probability the model assigns to the correct next token. Exponentiate it for perplexity, PPL = exp(L) — loosely, how many equally-likely tokens the model is choosing between at each step.
This gives two fixed reference points. At step zero, a correctly initialized model is near-uniform over its vocabulary, so the loss should start close to ln(V) — about ln(50000) ≈ 10.8 nats for a 50k vocabulary. Start far from there and something is wrong before training even begins. At the other end, the loss can never reach zero: natural language has irreducible entropy, so there is a floor L* no model can cross. Every healthy curve is a descent from ln(V) toward that floor, and the two endpoints tell you where on the journey you are.
Use log-log axes: the descent is a power law
On linear axes a training run looks like a cliff followed by a long, boring flat crawl, and you learn almost nothing from the flat part — a plotting artifact. Switch both axes to logarithmic and the same data straightens into a line, because loss falls as a power law in compute: L(C) ≈ L* + (C_0 / C)^α, where C is compute (steps or tokens) and L* is the irreducible floor.
Subtract the floor and plot L − L* against steps on log-log axes and a healthy run is a straight, downward line whose slope is the exponent α — the single most useful transform in the toolkit. A straight line means the run is on-trend and predictable: you can extrapolate where it will be in 10× the steps. A line that bends downward is a happy surprise; one that bends upward and flattens early means you are approaching your floor or starving the model of data or capacity. The log-log view converts ‘is the loss still going down?’ from a squint into a ruler.
What a healthy curve looks like
A healthy run has a recognizable silhouette. First a steep initial drop: in the first few hundred to few thousand steps the model learns the cheap, high-value structure — token frequencies, that spaces follow words, basic syntax — and the loss falls fast from near ln(V). Then the descent settles into the power-law regime, that steady straight line on log-log axes, each order of magnitude of compute buying a roughly constant fractional reduction in loss above the floor.
The line should be smooth at the scale of the schedule and only jittery at the scale of individual batches: high-frequency noise is healthy, but the trend should not wobble. Validation should track just above training and fall alongside it. If both descend on-trend with a thin, stable band and no widening gap, the run is boring — and boring is exactly what a good training run should be. Excitement in a loss curve is almost always bad news.
The warmup bump: reading the first few hundred steps
Zoom into the very start and you will often see the loss rise briefly, or dip and rebound, in the first tens to hundreds of steps. This bump is the fingerprint of learning-rate warmup, and it is expected, not a fault. Training ramps the learning rate from near zero to the peak over a warmup window precisely because a fresh, randomly initialized model plus a full-size learning rate is a recipe for an immediate blow-up.
During warmup the optimizer’s adaptive moment estimates (in Adam-family optimizers) are still calibrating on the first noisy gradients, so steps are erratic and the loss can jitter or tick up before the ramp reaches a stable rate. What to read is whether the bump resolves: a healthy warmup settles into the smooth descent by the time the ramp completes. A bump that instead grows into a full divergence is a different animal — the learning rate too high, below — and the tell is that it never recovers.
The noise floor: read the band, not the line
The training loss is never a clean line; it is a fuzzy band, and its thickness is itself a reading. Each step’s loss is measured on one batch, and batches genuinely differ in difficulty, so the point-to-point scatter reflects batch variance, not a change in the model. Larger batches average over more tokens, so the estimate is tighter and the band thinner; smaller batches give a fatter, noisier band. This is why a CPU-scale run with a tiny batch can look alarmingly jagged while being perfectly healthy.
The discipline is to read the band’s center for trend and its width for stability, never confusing the two: a single high point is one hard batch, not a problem. Apply a moving average if you must, but keep the raw band visible, because a sudden thickening — noise growing without the mean moving — is an early warning that stability is slipping, often the quiet prelude to a spike.
Divergence and spikes: the learning-rate-too-high signatures
The most dramatic shapes are spikes and divergence, and they usually mean the same thing: a step too aggressive for the loss landscape it landed in. A spike is a sudden jump in loss over a few steps followed by recovery to trend — a bad step, often on an unusually hard or malformed batch, clawed back. Occasional small spikes that self-heal are tolerable. Divergence is the unrecoverable version: the loss shoots up and stays up, collapsing toward NaN as gradients explode.
The classic signature of a learning rate set too high is a curve that descends encouragingly, then destabilizes — the band thickens, spikes grow more frequent, and eventually one fails to recover. Read the gradient norm alongside the loss: it almost always spikes first, so a jump in gradient norm with the loss still calm is the earliest tell. The steps are too big for the sharp regions of the landscape; a thickening band plus a leading gradient-norm spike confirm it.
Plateaus: flat is not always finished
A plateau is a stretch where the loss stops falling and runs flat, and the hard part is that several very different causes share the same appearance. The benign case is the end of the learning-rate schedule: as a cosine or linear decay drives the rate toward zero, progress slows and the curve flattens because the model is taking smaller and smaller steps by design. That is not a stall; it is the planned landing.
The malign cases look identical on linear axes, which is exactly why the log-log view matters. Flatten early, far above your floor and while the learning rate is still high, and you are likely underfit — too little capacity or data variety for the loss to keep dropping. Ask: where is the plateau relative to ln(V) and L*, and what is the learning rate doing now? A late, low, decaying-LR plateau is success; an early, high, full-LR one is a problem to chase.
The train-validation gap: reading generalization
Everything above reads a single curve for optimization health. The gap between the training and validation curves reads something else: generalization. Early on, the two lines sit almost on top of each other — the model is learning general structure that helps on any data. A small, stable gap with validation a touch above training is normal and fine.
The shape to fear is a widening gap where the training loss keeps falling while the validation loss flattens and then turns back up. That upturn is the textbook signature of overfitting: the model has started memorizing its training batches rather than learning transferable patterns, so it looks ever better on data it has seen and worse on data it has not. In large-scale near-single-epoch pretraining this classic overfitting is rare, but on the small models and repeated, limited datasets typical of CPU-scale work it is a live risk. The moment the validation curve turns up is the moment to stop, regularize, or get more data — the best checkpoint is the one at the validation minimum, not the last step.
Reading curves on CPU-scale runs
Small models on modest CPU budgets exaggerate every feature described here, which makes them a good place to learn to read but a bad place to panic. Tiny batches mean a fat, jagged band — jitter that would signal trouble on a large run is just the sampling noise of a 4k-token batch, so read the moving average and resist over-reacting to single points. Short runs show mostly the steep early drop and rarely reach a clean power-law regime, so extrapolating a slope from a few hundred steps is unreliable.
Two habits pay off here. Plot on log-log axes from the first run — it is the only way to tell an on-trend descent from an early underfit plateau when everything is compressed into a few thousand steps. And always log the validation curve, even cheaply, because the train-validation gap is where small-data, repeated-epoch CPU training gets you, and it is invisible on the training trace alone. A curve you can read beats a curve that merely ticks downward.