Why latent space: the arithmetic
Run diffusion on pixels and every one of your 30–50 denoising steps has to touch a 512×512×3 tensor. Stable Diffusion instead encodes the image once, diffuses in a small latent grid, and decodes once at the end. The saving is not marginal:
pixel tensor x: [B, 3, 512, 512] → 786,432 values
latent tensor z: [B, 4, 64, 64] → 16,384 values (48× fewer)
spatial positions: 512*512 = 262,144 → 64*64 = 4,096 (64× fewer)
self-attention pairs: 262,144^2 = 6.9e10 → 4,096^2 = 1.7e7 (4096×)Convolutional work scales with the number of spatial positions, so it drops by the downsampling factor squared, f^2 = 64. Attention scales as O(N^2) in positions, so it drops by f^4 = 4096. That second number is what makes the architecture possible at all: spatial self-attention over a quarter-million pixel positions is not something you run 60 times per image, but over 4,096 latent positions it is cheap. Latent diffusion buys the denoiser the right to use attention.
The autoencoder, and the 0.18215 you keep seeing
The compressor is a KL-regularised autoencoder trained separately and then frozen. The encoder E maps [B,3,512,512] → a mean and log-variance of shape [B,4,64,64]; you sample from it. The decoder D goes back. It is trained on reconstruction plus an LPIPS perceptual term plus a patch-GAN adversarial term, and — crucially — a KL weight around 1e-6. That weight is so small it is barely a VAE: the goal is faithful reconstruction, not a well-shaped prior, because the diffusion model supplies the prior.
But the raw latents come out with a standard deviation near 5.5, and the diffusion forward process assumes unit-scale data. So Stable Diffusion multiplies: z_0 = 0.18215 · E(x), and decodes D(z / 0.18215). The constant is just 1/σ measured on the training set. SDXL’s VAE has a different statistic, hence 0.13025.
The denoiser and what it is trained to output
The denoiser is a UNet operating entirely on [B,4,64,64]. It downsamples to 32, 16 and 8, with residual blocks at every level and transformer blocks at the 64, 32 and 16 levels plus the 8×8 mid block; only the deepest down/up stage is attention-free. Self-attention at the 64 level runs over 4,096 tokens — affordable only because of the f = 8 compression. The timestep enters by a different door from the text: a sinusoidal embedding of t is passed through a small MLP and added into every residual block, so the network knows how much noise to expect.
z_t = √(αbar_t) · z_0 + √(1 - αbar_t) · ε, ε ~ N(0, I)
L = E_[z_0, c, t, ε] || ε − ε_θ(z_t, t, c) ||^2One noise sample, one uniformly drawn t, one forward pass, one MSE. There is no per-timestep weighting in the shipped objective: dropping it already reweights the variational bound in a way that favours perceptual quality.
Cross-attention: where the prompt enters
The text is encoded once, outside the loop. SD 1.5 uses the CLIP ViT-L/14 text encoder, padded or truncated to exactly 77 tokens, giving a context c: [B, 77, 768]. SD 2 swaps in OpenCLIP ViT-H (d = 1024); SDXL concatenates two encoders to d = 2048. That tensor is then constant for the whole sampling run.
Inside each transformer block the spatial map is flattened to [B, hw, C] and hit twice: self-attention among image positions, then cross-attention where Q comes from the image and K, V come from c. The attention matrix is [B, hw, 77] — linear in image tokens, not quadratic. At the 32×32 level that is 1024 × 77 = 78,848 entries against self-attention’s 1024^2 = 1,048,576. Conditioning is nearly free; it is the image talking to itself that costs.
Classifier-free guidance: one network, two jobs
Guiding a diffusion model used to require a separately trained noisy-image classifier. Classifier-free guidance removes it with a one-line change to training: with probability p ≈ 0.1, throw away the caption and substitute the embedding of the empty string, ∅. The same weights therefore learn both the conditional and the unconditional denoiser. At sampling time you evaluate both and extrapolate along the difference:
ε_u = ε_θ(z_t, t, ∅) ε_c = ε_θ(z_t, t, c)
ε̂ = ε_u + w · (ε_c − ε_u)w = 0 is unconditional, w = 1 is plain conditional sampling, and w > 1 extrapolates past the conditional prediction — the default 7.5 pushes 6.5 steps beyond it. The bill is exact and unavoidable: two UNet evaluations per step, usually run as one batch of 2.
What the guidance scale does to the distribution
Because the noise prediction is proportional to the negative score of the noisy marginal, that extrapolation is not an ad-hoc image filter — it is sampling from a different distribution. The guided update is the score of p_w(z | c) ∝ p(z) · p(c | z)^w: the implicit classifier raised to the power w. Raising a likelihood to a power above 1 sharpens it, concentrating mass on the latents the model considers most unambiguously described by the prompt.
Hence the empirical trade every practitioner meets. Higher w raises prompt adherence (CLIP score) and lowers sample diversity and FID; the sweet spot for SD 1.5 sits around 6–9. Push past roughly 12 and the sharpening becomes distortion: the implied clean latent leaves the range the decoder was trained on, and you get blown highlights and hard contour edges. Two seeds at w = 20 also look far more alike than at w = 3 — the mode collapse is real.
DDIM: a deterministic update, and why 30 steps suffice
DDPM’s ancestral sampler injects fresh noise at every step and wants something like 1,000 of them. DDIM keeps the same trained network and the same marginals q(z_t | z_0) but defines a non-Markovian reverse process whose noise term you can dial to zero. First invert the forward equation for the implied clean latent, then re-noise it to the next timestep:
ẑ_0 = ( z_t − √(1 − αbar_t) · ε̂ ) / √(αbar_t)
z_(t−1) = √(αbar_(t−1)) · ẑ_0 + √(1 − αbar_(t−1)) · ε̂No random draw appears, so seed → image is a deterministic function. That is why you can subsample the schedule: take 30 of the 1,000 timesteps and the only error you accrue is discretisation error, with no injected noise that needs many steps to average out. It also makes the sampler invertible, which is what image editing and latent interpolation are built on.