Denoising diffusion probabilistic models (DDPMs, Ho, Jain & Abbeel 2020) generate data by learning to reverse a gradual corruption. A forward process slowly adds Gaussian noise to a clean sample over many steps until nothing is left but static; a reverse process, parameterized by a neural network, learns to undo one step of that noise at a time, so that starting from pure noise and denoising repeatedly produces a fresh sample from the data distribution. What makes DDPMs beautiful is that the forbidding-looking variational objective collapses, after the right change of variables, into something almost embarrassingly simple: predict the noise that was added, and score it with a plain mean-squared error. This piece builds that result from the ground up — the two Markov chains, the closed-form shortcuts, the bound they optimize, the epsilon-prediction trick, and the algorithms that fall out.

The two Markov chains

A DDPM is defined by two Markov chains over a sequence of latents x_0, x_1, …, x_T, all the same shape as the data. The forward (or diffusion) chain q is fixed, not learned: it takes a data point x_0 and adds a little Gaussian noise at each step until x_T is essentially N(0, I). The reverse chain p_θ is learned: it starts from noise x_T and removes noise step by step back to a clean x_0.

The whole design hinges on a single fact: if each forward step adds only a small amount of Gaussian noise, then each reverse step is also approximately Gaussian. So the network never has to model the full, wildly complicated data distribution in one shot — it only has to model a sequence of tiny Gaussian corrections. Breaking one impossible density-estimation problem into hundreds of easy ones is the core move that makes diffusion tractable, and everything below is machinery for training those tiny steps efficiently.

Advertisement

The forward process q

The forward process adds noise on a fixed variance schedule β_1, …, β_T with each β_t small (e.g. 10^-4 to 0.02):

q(x_t | x_{t-1}) = N( x_t ; sqrt(1 - β_t) · x_{t-1},  β_t · I )

The mean is the previous latent scaled down by sqrt(1 - β_t) and the added noise has variance β_t. That particular scaling is deliberate: it keeps the signal’s variance roughly constant as noise accumulates, so the chain converges to a standard Gaussian rather than drifting or blowing up. Because the whole chain is a product of Gaussians with fixed parameters, q has no learnable weights at all — it is a hand-specified corruption. The neural network appears only in the reverse direction. Sampling naively would mean applying this step t times to reach x_t; the next section removes that cost.

The closed-form marginal

Define α_t = 1 - β_t and the cumulative product ᾱ_t = ∏_{s=1}^{t} α_s. A key property of composing Gaussians is that the marginal q(x_t | x_0) is available in closed form — you can jump straight from the clean image to any noise level in one shot:

q(x_t | x_0) = N( x_t ; sqrt(ᾱ_t) · x_0,  (1 - ᾱ_t) · I )

x_t = sqrt(ᾱ_t) · x_0  +  sqrt(1 - ᾱ_t) · ε,     ε ~ N(0, I)

This reparameterization is what makes DDPM training cheap. To get a training example at step t you draw one Gaussian ε and mix it with x_0 using two scalar coefficients — no loop over t steps. As t → T, ᾱ_t → 0, so x_T loses essentially all of x_0 and becomes pure N(0, I), which is exactly the distribution the sampler will start from.

The reverse process p

Generation runs the chain backwards. We start from x_T ~ N(0, I) and define a learned reverse step as a Gaussian whose mean and (optionally) variance come from the network:

p_θ(x_{t-1} | x_t) = N( x_{t-1} ; μ_θ(x_t, t),  Σ_θ(x_t, t) )

The Gaussian form is justified by the small-step argument: when each forward step adds little noise, the true reverse conditional is close to Gaussian, so a Gaussian family is expressive enough. Ho et al. fix the covariance to an untrained, time-dependent scalar — Σ_θ = σ_t^2 I with σ_t^2 = β_t (or the posterior variance below) — so the network only has to predict the mean μ_θ. The same network is shared across all timesteps and told which t it is on via a positional-style time embedding, so one set of weights learns every denoising level.

The tractable posterior

Training needs a target for μ_θ. Although the reverse conditional q(x_{t-1} | x_t) is intractable, the posterior conditioned on the clean sample x_0 is a Gaussian we can write down exactly:

q(x_{t-1} | x_t, x_0) = N( x_{t-1} ; μ̃_t(x_t, x_0),  β̃_t · I )

μ̃_t = ( sqrt(ᾱ_{t-1}) · β_t / (1 - ᾱ_t) ) x_0
     + ( sqrt(α_t) (1 - ᾱ_{t-1}) / (1 - ᾱ_t) ) x_t

β̃_t = ( (1 - ᾱ_{t-1}) / (1 - ᾱ_t) ) · β_t

This falls out of Bayes’ rule combined with the closed-form marginals, and it is the linchpin of the whole derivation: it gives the reverse network an exact Gaussian target to match at every step. The training objective, developed next, simply asks p_θ(x_{t-1} | x_t) to stay close to this q(x_{t-1} | x_t, x_0).

The variational bound

DDPMs are latent-variable models, so they are trained by maximizing a variational lower bound on the log-likelihood (equivalently, minimizing an upper bound on the negative log-likelihood). Expanding the bound and grouping terms by timestep gives:

L = E_q[ D_KL( q(x_T|x_0) || p(x_T) )              ← L_T
       + Σ_{t>1} D_KL( q(x_{t-1}|x_t,x_0) || p_θ(x_{t-1}|x_t) )   ← L_{t-1}
       - log p_θ(x_0 | x_1) ]                    ← L_0

L_T has no learnable parameters — it just measures how close x_T is to N(0, I) and is essentially zero for a good schedule. L_0 is a reconstruction term for the final step. The heart of the objective is the sum of L_{t-1} terms, each a KL between two Gaussians — the tractable posterior and the network’s reverse step — which has a clean closed form and no Monte-Carlo estimation of the divergence.

From KL to a mean-matching loss

Because both distributions inside each L_{t-1} are Gaussians with the same fixed covariance σ_t^2 I, the KL divergence reduces to a scaled squared distance between their means:

L_{t-1} = E_q[ (1 / (2 σ_t^2)) · || μ̃_t(x_t, x_0) - μ_θ(x_t, t) ||^2 ] + C

All the machinery of variational inference has boiled down to a regression: make the network’s predicted mean match the posterior mean. This is already trainable, but it is not yet the elegant form Ho et al. are known for. The final step reparameterizes the mean in terms of the noise, which both simplifies the target and empirically improves sample quality — turning a mean-prediction problem into a noise-prediction problem.

Advertisement

Epsilon-prediction: the key simplification

Recall x_t = sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε. Solving for x_0 and substituting into μ̃_t shows the posterior mean depends on x_t and the noise ε that produced it. That motivates parameterizing the network to predict ε instead of the mean:

μ_θ(x_t, t) = (1 / sqrt(α_t)) · ( x_t - (β_t / sqrt(1 - ᾱ_t)) · ε_θ(x_t, t) )

Now the network ε_θ outputs a noise estimate the same shape as the data. Plugging this mean back into L_{t-1}, the coefficients collapse and the objective becomes a weighted MSE between the true noise ε and the predicted noise ε_θ. The task — ‘look at a noisy image and guess what noise was added’ — is intuitive and closely related to denoising score matching.

The simplified training objective

Ho et al. then make one pragmatic choice: drop the per-timestep weighting (the 1 / (2 σ_t^2) and remaining factors) and weight every timestep equally. This yields the celebrated simplified loss:

L_simple = E_{t, x_0, ε} [ || ε - ε_θ( sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε,  t ) ||^2 ]

with t drawn uniformly from {1, …, T} and ε ~ N(0, I). It is a plain mean-squared error on predicted noise — no KL, no variance terms, no likelihood bookkeeping in the code. Dropping the weights is not principled from the strict-bound view, but it emphasizes the harder, higher-noise steps and empirically produces markedly better samples. This single line is what most implementations actually optimize.

The training algorithm

The simplified loss gives a training loop of remarkable brevity — every iteration touches exactly one randomly chosen noise level:

repeat:
  x_0 ~ data
  t  ~ Uniform{1, ..., T}
  ε  ~ N(0, I)
  x_t = sqrt(ᾱ_t) · x_0 + sqrt(1 - ᾱ_t) · ε
  take gradient step on  ∇_θ || ε - ε_θ(x_t, t) ||^2
until converged

Note what is absent: there is no sequential unrolling of the chain during training. Thanks to the closed-form marginal, each step samples a single t and a single ε, builds x_t directly, and backpropagates through one network evaluation. Training cost per step is independent of T, so you can use T = 1000 steps at sampling time without paying for them during training — a major reason diffusion scaled so well.

The sampling algorithm

Generation is the reverse chain run one step at a time. Because each step needs a fresh network call, sampling is where DDPMs spend their compute:

x_T ~ N(0, I)
for t = T, T-1, ..., 1:
  z = N(0, I) if t > 1 else 0
  x_{t-1} = (1/sqrt(α_t)) · ( x_t - (β_t / sqrt(1 - ᾱ_t)) · ε_θ(x_t, t) )
           + σ_t · z
return x_0

Each iteration predicts the noise, subtracts a scaled version of it to get the posterior mean, then injects fresh noise σ_t z to keep the step stochastic — except the last step, which is deterministic so the final x_0 is clean. The stochastic injection is what makes DDPM a genuine probabilistic sampler; the sibling DDIM reformulation removes it to get a deterministic, faster path through the same trained model.

Noise schedules and step count

Two hyperparameters shape quality: the variance schedule {β_t} and the number of steps T. The original work used a linear schedule with T = 1000. Later work (Nichol & Dhariwal) showed a cosine schedule — which keeps ᾱ_t from collapsing too early — improves likelihood, especially at lower resolutions.

The schedule matters because it controls how ᾱ_t interpolates from 1 (clean) to 0 (pure noise). A poorly designed schedule wastes steps in regions where the image barely changes and starves the steps that do the real work. The cost of a large T is borne entirely at sampling time: 1000 steps means 1000 sequential network evaluations per sample, which is why so much follow-up research targets cutting that count without retraining.

Practical and compute implications

The asymmetry between training and sampling is the practical headline. Training is O(1) network calls per step and parallelizes cleanly across a batch of random (t, ε) pairs. Sampling is inherently sequential: step t-1 needs the output of step t, so a full-quality DDPM sample is hundreds to a thousand forward passes end to end — expensive on any hardware and punishing on a CPU.

This is exactly why so much engineering effort goes into sampling shortcuts. For CPU or latency-bound deployment you rarely run the vanilla 1000-step chain; you switch to a deterministic DDIM trajectory, a fast ODE/SDE solver, or a distilled few-step model — all of which reuse the same ε_θ weights trained by L_simple. Understanding vanilla DDPM first is what makes those accelerations legible.

A DDPM is two Markov chains: a fixed forward process that Gaussian-noises data to N(0, I), and a learned reverse process that denoises back. The closed-form marginal x_t = sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε lets you jump to any noise level in one shot, so training never unrolls the chain. The intimidating variational bound reduces — via the tractable posterior, a fixed covariance, and the epsilon-prediction reparameterization — to a single plain MSE: predict the noise you added, weight every timestep equally, and the network learns every denoising level at once. Training is cheap and parallel; sampling is the expensive part, a sequential walk of hundreds of network calls back from noise to a clean sample. That sequential cost is the price the whole family of faster samplers — DDIM and beyond — exists to cut, all on top of these same weights.