The curve is an instrument, not a scoreboard
Treating the loss curve as a scoreboard — one number, lower is better, glance and go — wastes it. It is closer to an oscilloscope: every hyperparameter leaves a signature in the trace, and once you know the signatures you can diagnose a run from the shape alone. The vertical axis is the loss (for language models, mean cross-entropy per token over a batch); the horizontal axis is progress, usually optimizer steps or tokens seen.
Two traces matter. The training loss, on the batch just learned from, is noisy and immediate. The validation loss, on held-out data the model never trains on, is smooth and honest. Reading well means reading both: the fast, jittery line for stability, the slow, clean line for generalization.
What the number actually is
Reading the shape starts with knowing what the height means. The loss is cross-entropy in nats (natural log) or bits (log base 2): L = -(1/N) Σ_i ln p(x_i), the average negative log-probability the model assigns to the correct next token. Exponentiate it for perplexity, PPL = exp(L) — loosely, how many equally-likely tokens the model is choosing between at each step.
This gives two fixed reference points. At step zero, a correctly initialized model is near-uniform over its vocabulary, so the loss should start close to ln(V) — about ln(50000) ≈ 10.8 nats for a 50k vocabulary. Start far from there and something is wrong before training even begins. At the other end, the loss can never reach zero: natural language has irreducible entropy, so there is a floor L* no model can cross. Every healthy curve is a descent from ln(V) toward that floor, and the two endpoints tell you where on the journey you are.
Use log-log axes: the descent is a power law
On linear axes a training run looks like a cliff followed by a long, boring flat crawl, and you learn almost nothing from the flat part — a plotting artifact. Switch both axes to logarithmic and the same data straightens into a line, because loss falls as a power law in compute: L(C) ≈ L* + (C_0 / C)^α, where C is compute (steps or tokens) and L* is the irreducible floor.
Subtract the floor and plot L − L* against steps on log-log axes and a healthy run is a straight, downward line whose slope is the exponent α — the single most useful transform in the toolkit. A straight line means the run is on-trend and predictable: you can extrapolate where it will be in 10× the steps. A line that bends downward is a happy surprise; one that bends upward and flattens early means you are approaching your floor or starving the model of data or capacity. The log-log view converts ‘is the loss still going down?’ from a squint into a ruler.
What a healthy curve looks like
A healthy run has a recognizable silhouette. First a steep initial drop: in the first few hundred to few thousand steps the model learns the cheap, high-value structure — token frequencies, that spaces follow words, basic syntax — and the loss falls fast from near ln(V). Then the descent settles into the power-law regime, that steady straight line on log-log axes, each order of magnitude of compute buying a roughly constant fractional reduction in loss above the floor.
The line should be smooth at the scale of the schedule and only jittery at the scale of individual batches: high-frequency noise is healthy, but the trend should not wobble. Validation should track just above training and fall alongside it. If both descend on-trend with a thin, stable band and no widening gap, the run is boring — and boring is exactly what a good training run should be. Excitement in a loss curve is almost always bad news.
The warmup bump: reading the first few hundred steps
Zoom into the very start and you will often see the loss rise briefly, or dip and rebound, in the first tens to hundreds of steps. This bump is the fingerprint of learning-rate warmup, and it is expected, not a fault. Training ramps the learning rate from near zero to the peak over a warmup window precisely because a fresh, randomly initialized model plus a full-size learning rate is a recipe for an immediate blow-up.
During warmup the optimizer’s adaptive moment estimates (in Adam-family optimizers) are still calibrating on the first noisy gradients, so steps are erratic and the loss can jitter or tick up before the ramp reaches a stable rate. What to read is whether the bump resolves: a healthy warmup settles into the smooth descent by the time the ramp completes. A bump that instead grows into a full divergence is a different animal — the learning rate too high, below — and the tell is that it never recovers.
The noise floor: read the band, not the line
The training loss is never a clean line; it is a fuzzy band, and its thickness is itself a reading. Each step’s loss is measured on one batch, and batches genuinely differ in difficulty, so the point-to-point scatter reflects batch variance, not a change in the model. Larger batches average over more tokens, so the estimate is tighter and the band thinner; smaller batches give a fatter, noisier band. This is why a CPU-scale run with a tiny batch can look alarmingly jagged while being perfectly healthy.
The discipline is to read the band’s center for trend and its width for stability, never confusing the two: a single high point is one hard batch, not a problem. Apply a moving average if you must, but keep the raw band visible, because a sudden thickening — noise growing without the mean moving — is an early warning that stability is slipping, often the quiet prelude to a spike.
Divergence and spikes: the learning-rate-too-high signatures
The most dramatic shapes are spikes and divergence, and they usually mean the same thing: a step too aggressive for the loss landscape it landed in. A spike is a sudden jump in loss over a few steps followed by recovery to trend — a bad step, often on an unusually hard or malformed batch, clawed back. Occasional small spikes that self-heal are tolerable. Divergence is the unrecoverable version: the loss shoots up and stays up, collapsing toward NaN as gradients explode.
The classic signature of a learning rate set too high is a curve that descends encouragingly, then destabilizes — the band thickens, spikes grow more frequent, and eventually one fails to recover. Read the gradient norm alongside the loss: it almost always spikes first, so a jump in gradient norm with the loss still calm is the earliest tell. The steps are too big for the sharp regions of the landscape; a thickening band plus a leading gradient-norm spike confirm it.