Classifier-Free Guidance (CFG)

The score-function view of diffusion

Diffusion sampling is easiest to reason about through the score: the gradient of the log-density, ∇_x log p(x). It points uphill toward regions of higher data probability, so a sampler that repeatedly nudges a noisy x along the score walks it from noise onto the data manifold. A trained noise-prediction network ε_θ(x_t, t) is exactly a scaled, sign-flipped score estimate:

ε_θ(x_t, t) ≈ -σ_t · ∇_x log p(x_t)

where σ_t is the noise level at step t. For conditional generation we sample from p(x | c) — images given a caption, audio given a speaker — so we need the conditional score ∇_x log p(x | c). Everything in CFG is about building that conditional score from pieces a single network can supply, so keep the score-vs-ε correspondence in mind: whatever we do to scores, we do the sign-flipped version to ε predictions.

Bayes rule and the implicit classifier

The conditional density factors by Bayes’ rule: p(x | c) ∝ p(x) · p(c | x). Take the log-gradient of both sides — the normalizer drops out because it does not depend on x:

∇_x log p(x | c) = ∇_x log p(x) + ∇_x log p(c | x)

The first term is the unconditional score (how to make any realistic sample); the second is the classifier gradient (how to make the sample more consistent with c). Classifier guidance literally trains a separate classifier p(c | x_t) and differentiates it. CFG’s insight is to rearrange the same equation to eliminate that classifier:

∇_x log p(c | x) = ∇_x log p(x | c) − ∇_x log p(x)

The classifier gradient equals the conditional score minus the unconditional score. If one network can produce both scores, we get the classifier for free — no auxiliary model, no noisy-image classifier to train, no separate gradients at sampling time.

Joint training with condition dropout

To supply both scores, CFG trains a single network to be conditional and unconditional at once. During training, with some probability p_uncond (commonly 10–20%), the condition is replaced by a special null token ∅ — an empty caption, a learned ‘no-condition’ embedding. The loss is the ordinary denoising objective:

L = E[ ‖ ε − ε_θ(x_t, t, c) ‖^2 ]   with c := ∅ w.p. p_uncond

The model thus learns ε_θ(x_t, t, c) for real conditions and ε_θ(x_t, t, ∅) for the marginal, sharing every weight; the condition enters through cross-attention or an added embedding, and passing ∅ switches the model into unconditional mode. No architectural change is needed — CFG is a training recipe plus a sampling formula, which is why it retrofits onto almost any conditional diffusion model.

The guidance equation

Now assemble the guided score. We want to sample not from p(x | c) but from a sharpened distribution p(x) · p(c | x)^s that raises the classifier term to a power s ≥ 1. Its score is:

∇ log p_s = ∇ log p(x) + s · ∇ log p(c | x)
        = ∇ log p(x) + s ( ∇ log p(x|c) − ∇ log p(x) )
        = (1 − s) ∇ log p(x) + s · ∇ log p(x|c)

Translating to noise predictions (flip the sign, absorb σ_t) gives the formula every sampler actually runs:

ε̃ = ε_θ(x_t, ∅) + s · ( ε_θ(x_t, c) − ε_θ(x_t, ∅) )

It is a straight-line extrapolation: start at the unconditional prediction, then step s times the vector pointing from unconditional toward conditional. At s = 1 it collapses to the plain conditional prediction; at s = 0 it is purely unconditional; at s > 1 it overshoots past the conditional prediction, exaggerating whatever the condition contributes.

Two conventions for the guidance scale

The literature uses two guidance parameters and they are easy to confuse. The interpolation form above uses a scale s where s = 1 is unguided. The original Ho & Salimans (2022) paper writes it with a weight w:

ε̃ = (1 + w) · ε_θ(x_t, c) − w · ε_θ(x_t, ∅)

Expand it and you find the identical line with s = 1 + w. So Ho’s w = 0 is unguided (pure conditional), whereas the guidance-scale s = 0 is unconditional. Stable Diffusion’s famous default guidance_scale = 7.5 is the s convention, equivalent to w = 6.5. When you read a formula, check which endpoint recovers the plain conditional model — that tells you which convention is in play. The algebra is the same; only the zero-point of the knob moves.

A worked numerical example

Take a single pixel-channel direction and pretend the network outputs 2-vectors. Suppose at some step ε_θ(x_t, c) = (2.0, −1.0) and ε_θ(x_t, ∅) = (1.5, −0.5). The guidance direction is:

d = ε(c) − ε(∅) = (2.0 − 1.5,  −1.0 − (−0.5)) = (0.5, −0.5)

With s = 7.5:

ε̃ = ε(∅) + 7.5 · d
   = (1.5, −0.5) + 7.5 · (0.5, −0.5)
   = (1.5 + 3.75,  −0.5 − 3.75) = (5.25, −4.25)

Notice how much larger the guided prediction is than either input: guidance did not average the two, it amplified their difference. The magnitude jumped from about 2.1 to about 6.8. That inflation is the whole mechanism — and, as we will see, also the source of CFG’s characteristic failure mode when s is pushed too high.

Why guidance sharpens the distribution

Raising p(c | x)^s to a power greater than one makes the classifier term peakier: samples that the implicit classifier considers strongly on-prompt get relatively far more probability mass, and ambiguous ones get suppressed. In distribution terms, guidance reduces variance and sharpens modes — it concentrates sampling onto the images that most decisively satisfy c.

This is exactly the fidelity-versus-diversity trade. Higher s yields images that adhere more tightly to the prompt and look crisper, at the cost of variety: outputs collapse toward a few canonical interpretations and rare valid compositions vanish. Lower s keeps samples diverse and natural but loosely conditioned. There is no universally correct value; you choose where to sit on a curve trading prompt adherence against sample entropy, depending on whether the task values obedience or exploration.

Plugging the guided score into a sampler

CFG changes only which noise estimate you feed the sampler; the DDPM or DDIM update rule is untouched. At each step you compute the two predictions, blend them with the guidance formula, and hand ε̃ to the ordinary update:

for t = T .. 1:
    e_c = eps(x_t, t, c)          # conditional pass
    e_u = eps(x_t, t, ∅)          # unconditional pass
    e   = e_u + s * (e_c - e_u)    # guided estimate
    x_{t-1} = sampler_step(x_t, e, t)

Because the guidance blend happens on the raw noise prediction, CFG is orthogonal to the choice of sampler — ancestral DDPM, deterministic DDIM, DPM-Solver, and friends all accept a guided ε without modification. Some implementations rescale ε̃ back toward the conditional prediction’s norm (‘guidance rescaling’) to fight the magnitude blow-up seen in the worked example, but the core loop is unchanged.

Shapes, batching, and the 2x cost

The price of CFG is stated in the loop above: two forward passes per step, one conditional and one unconditional, versus one for plain sampling. Naively that doubles inference cost. In practice you batch them — stack the conditional and null inputs into one tensor of shape [2B, C, H, W] and run a single forward pass, then split the output back into e_c and e_u:

x_in = concat([x_t, x_t])          # [2B, ...]
c_in = concat([c,   ∅])           # conditional + null
e_c, e_u = split( eps(x_in, t, c_in) )

Batching hides the latency behind parallelism on a GPU, but does not reduce the work: you still evaluate the network twice, doubling FLOPs and, in memory-bound settings, roughly doubling weight and activation traffic. The 2x factor is intrinsic to needing both scores — the single most important cost fact about CFG.

Choosing the guidance scale in practice

Typical values cluster by modality. Text-to-image diffusion commonly runs s ≈ 5–12 (Stable Diffusion defaults to 7.5); class-conditional models often sit lower; some tasks vary s over the trajectory — weaker early to preserve diversity, stronger later to lock in detail. Very high scales (s > 15) usually hurt: the amplified prediction pushes samples off the data manifold, producing oversaturated colors and blocky artifacts.

Two common repairs help. Dynamic thresholding clamps the predicted x_0 back into a valid range each step; guidance rescaling renormalizes ε̃ so its standard deviation matches the conditional prediction, keeping the direction of guidance but taming its inflated magnitude — a strong scale without the saturation tax.

Common pitfalls

The convention mix-up is the first trap: passing a Ho-style w where the code expects an s = 1 + w scale (or vice versa) silently under- or over-guides. Second, forgetting the null token at training time means the unconditional branch was never learned, so ε(∅) is garbage and guidance amplifies noise rather than signal. Third, the sign: because ε is a negative scaled score, it is easy to flip the extrapolation direction and end up steering away from the prompt.

Finally, ‘negative prompts’ are a use, not a bug, of the same math: replace the null branch ε(∅) with a prediction conditioned on an undesired prompt c−, and ε(c) − ε(c−) pushes toward the wanted concept and away from the unwanted one — same interpolation, different second endpoint.

What it means for CPU SLMs

On a CPU, where diffusion or conditional generation is already latency-bound, the 2x cost of CFG can be the difference between interactive and unusable. That makes it a prime target for cheap optimizations: batch the two passes so they share one matmul sweep of the weights (crucial when you are memory-bandwidth bound, as CPUs almost always are); or distill the guided model into a single conditional network via guidance distillation, training a student to reproduce ε̃ in one pass and erasing the 2x entirely.

The broader lesson generalizes past diffusion: CFG shows conditioning strength is a dial you set at inference, not a fixed property of the weights. One trained model spans the whole fidelity-diversity curve, and you pick the operating point per request.