Stable Diffusion is not a new theory of generative modelling — it is a set of engineering decisions stacked on top of one. The theory (a denoiser trained to invert Gaussian corruption) predates it. What Stable Diffusion added was the observation that you do not have to run that denoiser on pixels: you can run it on a compressed latent, condition it on text through cross-attention, and steer it at sampling time with a guidance trick that needs no classifier. Each choice is a concrete piece of arithmetic with a concrete cost, and each has a failure mode that shows up in the images. This piece walks the system: the shapes, the scale factor nobody explains, the guidance formula, the DDIM update, and where it breaks.
Why latent space: the arithmetic
Run diffusion on pixels and every one of your 30–50 denoising steps has to touch a 512×512×3 tensor. Stable Diffusion instead encodes the image once, diffuses in a small latent grid, and decodes once at the end. The saving is not marginal:
pixel tensor x: [B, 3, 512, 512] → 786,432 values
latent tensor z: [B, 4, 64, 64] → 16,384 values (48× fewer)
spatial positions: 512*512 = 262,144 → 64*64 = 4,096 (64× fewer)
self-attention pairs: 262,144^2 = 6.9e10 → 4,096^2 = 1.7e7 (4096×)Convolutional work scales with the number of spatial positions, so it drops by the downsampling factor squared, f^2 = 64. Attention scales as O(N^2) in positions, so it drops by f^4 = 4096. That second number is what makes the architecture possible at all: spatial self-attention over a quarter-million pixel positions is not something you run 60 times per image, but over 4,096 latent positions it is cheap. Latent diffusion buys the denoiser the right to use attention.
The autoencoder, and the 0.18215 you keep seeing
The compressor is a KL-regularised autoencoder trained separately and then frozen. The encoder E maps [B,3,512,512] → a mean and log-variance of shape [B,4,64,64]; you sample from it. The decoder D goes back. It is trained on reconstruction plus an LPIPS perceptual term plus a patch-GAN adversarial term, and — crucially — a KL weight around 1e-6. That weight is so small it is barely a VAE: the goal is faithful reconstruction, not a well-shaped prior, because the diffusion model supplies the prior.
But the raw latents come out with a standard deviation near 5.5, and the diffusion forward process assumes unit-scale data. So Stable Diffusion multiplies: z_0 = 0.18215 · E(x), and decodes D(z / 0.18215). The constant is just 1/σ measured on the training set. SDXL’s VAE has a different statistic, hence 0.13025.
The denoiser and what it is trained to output
The denoiser is a UNet operating entirely on [B,4,64,64]. It downsamples to 32, 16 and 8, with residual blocks at every level and transformer blocks at the 64, 32 and 16 levels plus the 8×8 mid block; only the deepest down/up stage is attention-free. Self-attention at the 64 level runs over 4,096 tokens — affordable only because of the f = 8 compression. The timestep enters by a different door from the text: a sinusoidal embedding of t is passed through a small MLP and added into every residual block, so the network knows how much noise to expect.
z_t = √(αbar_t) · z_0 + √(1 - αbar_t) · ε, ε ~ N(0, I)
L = E_[z_0, c, t, ε] || ε − ε_θ(z_t, t, c) ||^2One noise sample, one uniformly drawn t, one forward pass, one MSE. There is no per-timestep weighting in the shipped objective: dropping it already reweights the variational bound in a way that favours perceptual quality.
Cross-attention: where the prompt enters
The text is encoded once, outside the loop. SD 1.5 uses the CLIP ViT-L/14 text encoder, padded or truncated to exactly 77 tokens, giving a context c: [B, 77, 768]. SD 2 swaps in OpenCLIP ViT-H (d = 1024); SDXL concatenates two encoders to d = 2048. That tensor is then constant for the whole sampling run.
Inside each transformer block the spatial map is flattened to [B, hw, C] and hit twice: self-attention among image positions, then cross-attention where Q comes from the image and K, V come from c. The attention matrix is [B, hw, 77] — linear in image tokens, not quadratic. At the 32×32 level that is 1024 × 77 = 78,848 entries against self-attention’s 1024^2 = 1,048,576. Conditioning is nearly free; it is the image talking to itself that costs.
Classifier-free guidance: one network, two jobs
Guiding a diffusion model used to require a separately trained noisy-image classifier. Classifier-free guidance removes it with a one-line change to training: with probability p ≈ 0.1, throw away the caption and substitute the embedding of the empty string, ∅. The same weights therefore learn both the conditional and the unconditional denoiser. At sampling time you evaluate both and extrapolate along the difference:
ε_u = ε_θ(z_t, t, ∅) ε_c = ε_θ(z_t, t, c)
ε̂ = ε_u + w · (ε_c − ε_u)w = 0 is unconditional, w = 1 is plain conditional sampling, and w > 1 extrapolates past the conditional prediction — the default 7.5 pushes 6.5 steps beyond it. The bill is exact and unavoidable: two UNet evaluations per step, usually run as one batch of 2.
What the guidance scale does to the distribution
Because the noise prediction is proportional to the negative score of the noisy marginal, that extrapolation is not an ad-hoc image filter — it is sampling from a different distribution. The guided update is the score of p_w(z | c) ∝ p(z) · p(c | z)^w: the implicit classifier raised to the power w. Raising a likelihood to a power above 1 sharpens it, concentrating mass on the latents the model considers most unambiguously described by the prompt.
Hence the empirical trade every practitioner meets. Higher w raises prompt adherence (CLIP score) and lowers sample diversity and FID; the sweet spot for SD 1.5 sits around 6–9. Push past roughly 12 and the sharpening becomes distortion: the implied clean latent leaves the range the decoder was trained on, and you get blown highlights and hard contour edges. Two seeds at w = 20 also look far more alike than at w = 3 — the mode collapse is real.
DDIM: a deterministic update, and why 30 steps suffice
DDPM’s ancestral sampler injects fresh noise at every step and wants something like 1,000 of them. DDIM keeps the same trained network and the same marginals q(z_t | z_0) but defines a non-Markovian reverse process whose noise term you can dial to zero. First invert the forward equation for the implied clean latent, then re-noise it to the next timestep:
ẑ_0 = ( z_t − √(1 − αbar_t) · ε̂ ) / √(αbar_t)
z_(t−1) = √(αbar_(t−1)) · ẑ_0 + √(1 − αbar_(t−1)) · ε̂No random draw appears, so seed → image is a deterministic function. That is why you can subsample the schedule: take 30 of the 1,000 timesteps and the only error you accrue is discretisation error, with no injected noise that needs many steps to average out. It also makes the sampler invertible, which is what image editing and latent interpolation are built on.
Noise schedules and the terminal-SNR flaw
The schedule is the sequence β_1..β_T with αbar_t = ∏(1 − β_s). Stable Diffusion uses scaled-linear: linear in √β from 0.00085 to 0.012 over T = 1000, which spends more capacity at low noise than DDPM’s plain linear ramp.
It also has a well-documented bug. That schedule ends at αbar_T ≈ 0.0047, so √(αbar_T) ≈ 0.068 — about 7% of the original latent, including its mean brightness, survives to the final step. Training therefore never shows the model a genuinely pure Gaussian, but inference starts from exactly that. The model leaks the training set’s average luminance, which is why vanilla SD struggles to produce a truly black or truly white image. The fix is a rescaled schedule with zero terminal SNR, v-prediction (which stays well-conditioned as αbar → 0), and trailing timestep spacing.
The pipeline and where the milliseconds go
Assemble it: encode the prompt and the empty prompt; draw z_T ~ N(0, I) of shape [1,4,64,64]; loop S steps, each a batch-2 UNet pass, a CFG combine, and a scheduler update; then decode once. For S = 30 on a mid-range GPU the shape of the budget is roughly:
| Stage | Forward passes | Share of wall time |
|---|---|---|
| CLIP text encode | 1 (batch 2) | ~1% |
| UNet sampling loop | 60 | ~90% |
| VAE decode | 1 | ~9% |
Two consequences. The decoder is a single pass but runs at full 512×512, so it is not free. And step count is the only exactly linear knob — halving S halves the run. On CPU, where a UNet pass is seconds rather than milliseconds, those 60 passes are the whole reason latency is measured in minutes, and why distilled few-step models matter far more there than on a GPU.
Failure modes
The most under-appreciated one is the reconstruction ceiling. No sampler can beat D(E(x)), and an f = 8 autoencoder discards genuine high-frequency detail. Round-trip a photograph through the VAE with no diffusion at all and you will see small faces smear, thin lines wobble and text turn to pseudo-glyphs. That is the origin of SD’s famous inability to write, and it is why img2img chains degrade — every pass pays the encode/decode toll again.
The rest are cheaper to diagnose. Contrast-fried output means w is too high; guidance rescaling recovers it. Grey or noisy garbage usually means a mismatched scale factor (SDXL latents divided by 0.18215). Resolutions that are not multiples of 8 get silently floored in the latent grid. And prompts far outside the training distribution make the two CFG branches disagree wildly, so guidance extrapolates straight off the manifold.
f = 8 autoencoder cuts spatial positions 64× and attention cost 4096×, which is what makes transformer blocks affordable inside the UNet — at the price of a hard reconstruction ceiling that no sampler can beat, and a scale factor (0.18215) that must match the VAE. Cross-attention injects a frozen [B,77,d] text context at linear cost in image tokens. Classifier-free guidance buys prompt adherence by sampling from p(z) · p(c|z)^w, doubling the per-step cost and trading diversity for fidelity until it over-sharpens. The sampler is a swap, not a retrain: DDIM’s deterministic update lets 30 steps stand in for 1,000, and almost all the latency is those 2S UNet passes.