Diffusion Models
What Diffusion Models Do
Diffusion models generate images, audio, video, 3D structures, and scientific data by turning random noise into a structured sample through repeated denoising.
They are especially useful when sample quality, diversity, and stable training matter. Text-to-image systems use conditioning to guide the denoising process toward a prompt or another input.
Core Idea
Training starts with real data and progressively corrupts it with Gaussian noise. A neural network learns the reverse operation: given a noisy sample and its noise level, predict the noise that should be removed.
Generation runs that learned process backward. It begins at nearly pure Gaussian noise and takes a sequence of small denoising steps until a new sample appears.
An intuitive analogy is ink dispersing in water. The forward process records how structure disappears; the model learns how to reconstruct plausible structure from each degree of corruption.
How It Works
1. Define the Forward Process
Start with a clean sample x_0. At each time step t, add a controlled amount of Gaussian noise. After T steps, commonly 1,000 in early systems, x_T is approximately N(0, I).
x_0 → x_1 → x_2 → ... → x_T ≈ N(0, I)
A variance schedule β_1, …, β_T determines how quickly information is destroyed. Linear schedules are simple; cosine schedules preserve more signal during early steps.
The forward process is fixed rather than learned. Using cumulative signal retention ᾱ_t, any noisy state can be sampled directly:
x_t = sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε, where ε ~ N(0, I)
This closed form avoids running all preceding noising steps during training.
2. Train a Noise Predictor
A U-Net or transformer receives x_t, the time step t, and optional conditioning. It predicts the sampled noise ε with a mean-squared error objective:
L = ||ε - ε_θ(x_t, t)||²
Time embeddings tell the network how corrupted its input is. U-Net skip connections preserve spatial detail, while cross-attention can inject text, image, or other conditioning.
| Conditioning | Typical mechanism | Example use |
|---|---|---|
| Text | Cross-attention with text embeddings | Text-to-image |
| Class | Add a learned class embedding | Class-conditional synthesis |
| Image | Concatenation or cross-attention | Editing and image-to-image |
| None | No conditioning signal | Unconditional generation |
3. Sample in Reverse
Sampling begins with x_T ~ N(0, I). At each reverse step, the network estimates noise or an equivalent parameterization, and the sampler computes a slightly cleaner x_{t-1}.
x_T (noise) → x_{T-1} → ... → x_1 → x_0 (generated sample)
The model solves many local denoising problems instead of producing the complete sample in one leap. The exact update depends on whether the sampler is stochastic, deterministic, or formulated as an ODE/SDE solver.
4. Guide the Result
Conditional models learn how data and conditioning co-vary. Classifier-free guidance combines conditional and unconditional predictions to strengthen alignment, trading some diversity for closer adherence.
Latent diffusion first compresses images with an autoencoder. Denoising in the smaller latent space reduces memory and computation, and a decoder maps the final latent back to pixels.
5. Accelerate Sampling
- DDIM follows a deterministic trajectory and can use fewer steps.
- Latent diffusion moves expensive computation into a compressed representation.
- Improved numerical solvers take larger, more accurate reverse steps.
- Consistency models distill iterative generation toward one or a few steps.
Why It Works
- Simple supervision: Known noise provides a target for every corrupted training sample.
- Stable optimization: One network solves a regression problem rather than a two-player adversarial game.
- Multi-scale learning: Heavy noise teaches global structure; lighter noise teaches textures and details.
- Broad mode coverage: The likelihood-based formulation is less prone to dropping data modes than GAN training.
- Flexible conditioning: Attention and guidance can steer generation without changing the basic denoising framework.
- Scalable backbones: U-Nets and transformers can absorb more data, parameters, and compute effectively.
Where They Are Used
- Images: text-to-image generation, inpainting, super-resolution, style transfer, and editing.
- Video: joint spatial and temporal denoising.
- Audio: waveform or spectrogram synthesis.
- 3D: point clouds and NeRF-related generation.
- Science: molecular design and protein structure generation.
Comparison with Other Generators
| Model | Main strength | Main weakness |
|---|---|---|
| GAN | One-pass sampling | Unstable adversarial training and mode collapse |
| VAE | Stable training and useful latent space | Often blurry under pixel-wise reconstruction losses |
| Diffusion | High quality, diversity, and stable training | Iterative sampling is expensive |
| Flow-based | Exact likelihood | Restrictive invertible architecture |
Compared with GANs, diffusion models generally cover modes better and expose a likelihood-related objective. GANs remain attractive when very low sampling latency is more important than iterative refinement.
Limitations
- Slow generation: Many network evaluations may be required for one sample.
- High compute cost: Large denoisers are expensive to train and serve, especially in pixel or video space.
- Guidance trade-offs: Strong guidance can reduce diversity, oversaturate images, or amplify artifacts.
- Data dependence: Outputs can reproduce dataset biases, copyrighted styles, unsafe content, or memorized details.
- Limited control: Prompts do not guarantee precise spatial relationships or factual structure.
- Compression loss: Latent diffusion inherits detail limits and artifacts from its autoencoder.
- Evaluation gaps: Automated scores do not fully capture prompt alignment, originality, or human preference.