Classifier Guidance Math

What classifier guidance is

A diffusion model generates by starting from pure noise x_T and denoising step by step down to a clean sample x_0. A plain unconditional model wanders toward any plausible image. Classifier guidance hijacks that walk: at every step it asks a classifier ‘how would I have to change this half-denoised image to make you more confident it is class y?’ and pushes the sample a little in that direction.

Concretely, you need two models. First, an ordinary diffusion model ε_θ(x_t, t) that predicts the noise in a corrupted image. Second, a classifier p_φ(y | x_t) that reads a noisy image at timestep t and outputs class probabilities. Sampling proceeds as normal, except each step is perturbed by the classifier’s gradient with respect to the image. The diffusion model supplies ‘what looks realistic,’ the classifier supplies ‘what looks like a y,’ and guidance is the recipe for combining the two.

The score-function view of diffusion

The cleanest way to understand guidance is through the score — the gradient of the log-density of the data with respect to the image, ∇_x log p(x_t). A trained diffusion model is, up to a scale factor, a score estimator. If x_t = √(α̅_t) x_0 + √(1 − α̅_t) ε, then the noise the model predicts is tied to the score by:

∇_x_t log p(x_t)  ≈  − ε_θ(x_t, t) / σ_t

where  σ_t = √(1 − α̅_t)   (the noise std at step t)

So predicting noise and estimating the score are the same job wearing different clothes. This matters because a sampler — Langevin dynamics, DDPM, DDIM — only ever consumes the score. If we can write down the score of the conditional distribution p(x_t | y), we can feed it to the exact same sampler and get class-conditional samples for free. Guidance is the derivation of that conditional score.

Bayes’ rule splits the conditional score

Start from Bayes’ rule for the conditional density at noise level t:

p(x_t | y)  =  p(x_t) · p(y | x_t) / p(y)

Take the log, then the gradient with respect to x_t. The term log p(y) does not depend on x_t, so its gradient is zero and it drops out entirely:

∇_x_t log p(x_t | y)  =  ∇_x_t log p(x_t)  +  ∇_x_t log p(y | x_t)

     conditional score   =   unconditional score   +   classifier gradient

That single line is all of classifier guidance. The conditional score you want is the unconditional score you already have (from the diffusion model) plus one extra term: the gradient of the classifier’s log-probability for the target class. The diffusion model never has to be retrained per class — the class information is injected entirely through that additive gradient.

The gradient ∇ log p(y|x_t)

The new ingredient is ∇_x_t log p_φ(y | x_t): how the classifier’s confidence in class y changes as we perturb each pixel. It is exactly the quantity an adversarial-example attack computes — the input gradient that most increases the score for a chosen label — obtained by one backward pass through the classifier. Moving x_t a small step along this gradient makes the image look more like a y to the classifier.

Shape-wise it lives in the same space as the image: if x_t is [C, H, W], the gradient is [C, H, W], one number per pixel-channel. Each sampling step therefore requires a forward and backward pass through the classifier, in addition to the diffusion model’s forward pass. The key subtlety — the thing that makes classifier guidance more than a textbook identity — is that this gradient must be evaluated at the noisy x_t, not a clean image. An off-the-shelf ImageNet classifier would be useless here; it has never seen an image buried in Gaussian noise.

Training a noise-robust classifier

This is the ingredient that makes classifier guidance expensive and awkward, and it is what most distinguishes it from its successor. You cannot reuse a standard classifier. You must train a bespoke one that accepts the timestep t and is fed images corrupted to every noise level the sampler will visit.

The recipe: sample a clean image x_0 and a label y from the dataset, sample a random timestep t, form the noised x_t with the forward diffusion process, and train p_φ(y | x_t, t) with ordinary cross-entropy. At small t the image is nearly clean and the task is easy; at large t the image is almost pure noise and the classifier can do little better than the class prior. In practice the encoder half of a U-Net (with timestep conditioning) is used as the classifier backbone. This is a whole extra network to build, train, and store — and its gradients drive the sampler, so its quality directly caps the guided model’s fidelity.

The guidance scale

Using the raw gradient gives correct conditional samples but often weak class adherence. In practice you multiply the classifier term by a guidance scale s ≥ 1:

guided score  =  ∇_x_t log p(x_t)  +  s · ∇_x_t log p(y | x_t)

Setting s = 1 recovers the exact Bayes-correct conditional. Setting s > 1 is equivalent to sampling from a sharpened classifier distribution p(y | x_t)^s, which concentrates probability on the most unambiguously class-y regions. The effect is the fidelity–diversity knob at the heart of guided diffusion: raise s and every sample screams ‘golden retriever,’ but the outputs collapse toward a few canonical, high-confidence modes and lose variety; lower s and you get diverse but sometimes off-target images. Dhariwal and Nichol found modest scales bought large gains in the class-conditional metrics (FID and Inception Score) that GANs had dominated — the result that put diffusion on the map.

Plugging it into the sampler

Most samplers are written in terms of the predicted noise, so convert the score back. Because ∇ log p = −ε/σ_t, adding s · ∇ log p(y|x_t) to the score is the same as subtracting a scaled gradient from the predicted noise:

ε̂(x_t, t)  =  ε_θ(x_t, t)  −  s · σ_t · ∇_x_t log p_φ(y | x_t)

The DDPM view is equivalent and often quoted as a shift of the sampling mean: the reverse step draws x_{t-1} from a Gaussian whose mean is nudged by s · Σ · ∇_x_t log p_φ(y | x_t), where Σ is that step’s covariance. Either way the per-step loop becomes: (1) run the diffusion model for ε_θ; (2) run the classifier and backprop to get the label gradient; (3) combine them into ε̂; (4) take the ordinary reverse-diffusion step with the corrected noise. Nothing about the sampler itself changes — only the noise it is handed.

A worked numeric example

Take a single pixel-channel to keep the arithmetic visible. Suppose at some step σ_t = 0.5, the diffusion model predicts noise ε_θ = 0.20, and the classifier’s log-probability gradient for the target class at that pixel is ∇ log p(y|x_t) = 0.80 (the classifier ‘wants’ this pixel brighter to look more like y).

s = 1:   ε̂ = 0.20 − 1 × 0.5 × 0.80 = 0.20 − 0.40 = −0.20
s = 3:   ε̂ = 0.20 − 3 × 0.5 × 0.80 = 0.20 − 1.20 = −1.00

At s = 1 the effective noise flips sign and shrinks, a gentle steer. At s = 3 it swings strongly negative — the sampler removes much more ‘noise’ in the direction that brightens the pixel, pushing hard toward the class. You can feel the trade directly: tripling s did not gently adjust the update, it dominated the diffusion model’s own estimate. Push it too far and guidance overwhelms realism, giving over-saturated, artifact-ridden images that satisfy the classifier but no longer look natural.

Why it works, and what it costs

Guidance works because it decomposes an intractable object — the class-conditional data score — into two things we can each estimate: an unconditional generative model and a discriminative classifier. Neither has to solve the whole problem. It is a clean division of labour, and it means one unconditional diffusion model can be steered toward any label for which you can train a classifier, without touching the generator.

The costs are real, though. You maintain and train a second network, and every sampling step pays an extra forward and backward pass through it, so guided runs get meaningfully slower. The classifier must be noise-aware, which rules out the huge pretrained models the community already owns. And the signal is only as good as the classifier’s gradients — a poorly calibrated or fragile classifier can steer the sampler into images that fool it without looking like the class at all.

The limitations that motivated CFG

Two problems in particular pushed the field past classifier guidance. First is the separate-classifier tax: needing a distinct, noise-robust classifier with its own training run and storage doubles the engineering surface and couples the generator’s quality to a second model’s. Second, and more subtle, is adversarial gradient hacking. The sampler optimizes the image to maximize a classifier’s score, which is precisely the setup that produces adversarial examples; high guidance can exploit classifier blind spots and yield images that score highly for y yet look wrong.

Classifier-free guidance (Ho and Salimans, 2022) answers both. It trains a single model jointly on the conditional and unconditional objectives (by randomly dropping the condition) and reconstructs the same guidance direction as the difference ε_θ(x_t, y) − ε_θ(x_t, ∅) — no external classifier, no separate gradient, no adversarial handle. That is why virtually every modern text-to-image system uses CFG. Classifier guidance remains the conceptual foundation: CFG is the same Bayes decomposition with the classifier folded into the diffusion model itself.

What it means for CPU SLMs

For small models running on CPU, the lesson is about where the compute goes. Classifier guidance turns one network evaluation per step into two, and the second carries a backward pass — roughly a 2–3× step cost that a CPU feels acutely, since diffusion sampling is already dozens to hundreds of sequential steps. On constrained hardware that overhead is often the difference between usable and unusable latency.

The takeaways carry to any conditioned pipeline you run locally. Prefer classifier-free guidance where you can: no extra network and no sample-time gradient make it strictly cheaper per step. Keep the guidance scale modest — the sharpening is nonlinear and the sweet spot is usually small. And remember the identity is timestep-aware: condition on the noisy state, not the clean one. That single insight is what classifier guidance taught the whole field.