Diffusion Models

What Diffusion Models Do

Diffusion models generate images, audio, video, 3D structures, and scientific data by turning random noise into a structured sample through repeated denoising.

They are especially useful when sample quality, diversity, and stable training matter. Text-to-image systems use conditioning to guide the denoising process toward a prompt or another input.

Core Idea

Training starts with real data and progressively corrupts it with Gaussian noise. A neural network learns the reverse operation: given a noisy sample and its noise level, predict the noise that should be removed.

Generation runs that learned process backward. It begins at nearly pure Gaussian noise and takes a sequence of small denoising steps until a new sample appears.

An intuitive analogy is ink dispersing in water. The forward process records how structure disappears; the model learns how to reconstruct plausible structure from each degree of corruption.

How It Works

1. Define the Forward Process

Start with a clean sample x_0. At each time step t, add a controlled amount of Gaussian noise. After T steps, commonly 1,000 in early systems, x_T is approximately N(0, I).

x_0 → x_1 → x_2 → ... → x_T ≈ N(0, I)

A variance schedule β_1, …, β_T determines how quickly information is destroyed. Linear schedules are simple; cosine schedules preserve more signal during early steps.

The forward process is fixed rather than learned. Using cumulative signal retention ᾱ_t, any noisy state can be sampled directly:

x_t = sqrt(ᾱ_t) x_0 + sqrt(1 - ᾱ_t) ε,  where ε ~ N(0, I)

This closed form avoids running all preceding noising steps during training.

2. Train a Noise Predictor

A U-Net or transformer receives x_t, the time step t, and optional conditioning. It predicts the sampled noise ε with a mean-squared error objective:

L = ||ε - ε_θ(x_t, t)||²

Time embeddings tell the network how corrupted its input is. U-Net skip connections preserve spatial detail, while cross-attention can inject text, image, or other conditioning.

Conditioning Typical mechanism Example use
Text Cross-attention with text embeddings Text-to-image
Class Add a learned class embedding Class-conditional synthesis
Image Concatenation or cross-attention Editing and image-to-image
None No conditioning signal Unconditional generation

3. Sample in Reverse

Sampling begins with x_T ~ N(0, I). At each reverse step, the network estimates noise or an equivalent parameterization, and the sampler computes a slightly cleaner x_{t-1}.

x_T (noise) → x_{T-1} → ... → x_1 → x_0 (generated sample)

The model solves many local denoising problems instead of producing the complete sample in one leap. The exact update depends on whether the sampler is stochastic, deterministic, or formulated as an ODE/SDE solver.

4. Guide the Result

Conditional models learn how data and conditioning co-vary. Classifier-free guidance combines conditional and unconditional predictions to strengthen alignment, trading some diversity for closer adherence.

Latent diffusion first compresses images with an autoencoder. Denoising in the smaller latent space reduces memory and computation, and a decoder maps the final latent back to pixels.

5. Accelerate Sampling

  • DDIM follows a deterministic trajectory and can use fewer steps.
  • Latent diffusion moves expensive computation into a compressed representation.
  • Improved numerical solvers take larger, more accurate reverse steps.
  • Consistency models distill iterative generation toward one or a few steps.

Why It Works

  • Simple supervision: Known noise provides a target for every corrupted training sample.
  • Stable optimization: One network solves a regression problem rather than a two-player adversarial game.
  • Multi-scale learning: Heavy noise teaches global structure; lighter noise teaches textures and details.
  • Broad mode coverage: The likelihood-based formulation is less prone to dropping data modes than GAN training.
  • Flexible conditioning: Attention and guidance can steer generation without changing the basic denoising framework.
  • Scalable backbones: U-Nets and transformers can absorb more data, parameters, and compute effectively.

Where They Are Used

  • Images: text-to-image generation, inpainting, super-resolution, style transfer, and editing.
  • Video: joint spatial and temporal denoising.
  • Audio: waveform or spectrogram synthesis.
  • 3D: point clouds and NeRF-related generation.
  • Science: molecular design and protein structure generation.

Comparison with Other Generators

Model Main strength Main weakness
GAN One-pass sampling Unstable adversarial training and mode collapse
VAE Stable training and useful latent space Often blurry under pixel-wise reconstruction losses
Diffusion High quality, diversity, and stable training Iterative sampling is expensive
Flow-based Exact likelihood Restrictive invertible architecture

Compared with GANs, diffusion models generally cover modes better and expose a likelihood-related objective. GANs remain attractive when very low sampling latency is more important than iterative refinement.

Limitations

  • Slow generation: Many network evaluations may be required for one sample.
  • High compute cost: Large denoisers are expensive to train and serve, especially in pixel or video space.
  • Guidance trade-offs: Strong guidance can reduce diversity, oversaturate images, or amplify artifacts.
  • Data dependence: Outputs can reproduce dataset biases, copyrighted styles, unsafe content, or memorized details.
  • Limited control: Prompts do not guarantee precise spatial relationships or factual structure.
  • Compression loss: Latent diffusion inherits detail limits and artifacts from its autoencoder.
  • Evaluation gaps: Automated scores do not fully capture prompt alignment, originality, or human preference.

Related