GAN (Generative Adversarial Network)
What a GAN Does
A generative adversarial network learns to create new samples that resemble its training data. GANs have generated images, video, audio, and 3D content, often in a single fast inference pass.
Introduced in 2014 in Generative Adversarial Nets, the method replaced a hand-designed similarity loss with an adaptive opponent. It dominated image synthesis from roughly 2015 to 2021.
DCGAN (2015), Pix2Pix (2016), CycleGAN (2017), StyleGAN (2018), BigGAN (2018), and StyleGAN2 (2020) progressively improved stability, control, scale, and visual fidelity.
Core Idea
A GAN trains two neural networks in competition:
- The generator G maps random noise
zto a synthetic sampleG(z). - The discriminator D estimates whether a sample came from the real data rather than the generator.
- The generator improves by learning which changes make the discriminator more likely to accept its output.
The familiar analogy is a counterfeiter and a detective: each becomes better in response to the other. At the ideal equilibrium, generated and real distributions match, and the discriminator outputs 0.5 everywhere.
GAN: Generator vs. Discriminator adversarial training
Source: Goodfellow et al., 2014 — Generative Adversarial Nets (NeurIPS)
How It Works
1. Generate a Candidate
Sample a latent vector z, commonly from a Gaussian distribution. The generator transforms it through learned layers into an output with the same structure as the training data.
A typical DCGAN generator expands a 100-dimensional vector into a 64 × 64 × 3 image:
| Layer | Operation | Output |
|---|---|---|
| Input | Noise vector z |
100 |
| Projection | Linear + reshape | 4 × 4 × 1024 |
| Block 1 | ConvTranspose 4×4/2 + BN + ReLU |
8 × 8 × 512 |
| Block 2 | ConvTranspose 4×4/2 + BN + ReLU |
16 × 16 × 256 |
| Block 3 | ConvTranspose 4×4/2 + BN + ReLU |
32 × 32 × 128 |
| Output | ConvTranspose 4×4/2 + Tanh |
64 × 64 × 3 |
2. Judge Real and Generated Samples
The discriminator is usually a downsampling CNN. It processes either a real or generated image and returns one real/fake score.
| Layer | Operation | Output |
|---|---|---|
| Input | Real or generated image | 64 × 64 × 3 |
| Block 1 | Conv 4×4/2 + LeakyReLU |
32 × 32 × 128 |
| Block 2 | Conv 4×4/2 + BN + LeakyReLU |
16 × 16 × 256 |
| Block 3 | Conv 4×4/2 + BN + LeakyReLU |
8 × 8 × 512 |
| Block 4 | Conv 4×4/2 + BN + LeakyReLU |
4 × 4 × 1024 |
| Output | Flatten + linear + Sigmoid | 1 |
LeakyReLU helps prevent dead discriminator units. Batch normalization stabilizes both networks, except at the discriminator input and generator output.
3. Optimize the Minimax Objective
The original value function is:
min_G max_D V(D, G) = E[log D(x)] + E[log(1 - D(G(z)))]
The discriminator maximizes correct decisions on both real x and generated G(z). The generator minimizes rejection; in practice, the non-saturating objective maximizes log D(G(z)) to provide stronger gradients.
Training alternates discriminator and generator updates. Neither objective is stationary because every update changes the opponent that defines the other network’s learning signal.
4. Condition or Modify the Game
A conditional GAN gives both networks a signal y, such as a class, text prompt, or source image. The pair G(z, y) and D(x, y) enables controlled generation and image-to-image translation.
Wasserstein GAN replaces the original Jensen–Shannon-based game with Earth Mover distance. Weight clipping or a gradient penalty enforces the required Lipschitz constraint and makes the training signal more useful.
Why It Works
- A learned loss: The discriminator discovers discrepancies that a fixed pixel loss may miss.
- Implicit distribution learning: The generator need not define or evaluate an explicit probability density.
- Perceptual sharpness: Adversarial feedback rewards realistic structure instead of averaging plausible outputs into blur.
- Competitive refinement: Better generated samples force a stronger discriminator, which can provide finer feedback in return.
- Fast sampling: Once trained, generation is one forward pass rather than an iterative procedure.
This adversarial principle influenced perceptual losses, image restoration, super-resolution, representation learning, self-supervised learning, and adversarial robustness research.
Important Variants
| Model | Year | Main contribution |
|---|---|---|
| GAN | 2014 | Original adversarial framework |
| cGAN | 2014 | Generation conditioned on labels or other inputs |
| DCGAN | 2015 | Convolutional baseline with batch normalization |
| Pix2Pix | 2016 | Paired image-to-image translation |
| CycleGAN | 2017 | Unpaired translation through cycle consistency |
| WGAN | 2017 | Wasserstein distance for more stable gradients |
| StyleGAN | 2018 | Style-based generator and intermediate latent space |
| BigGAN | 2018 | Large-scale class-conditional generation |
| StyleGAN2 | 2020 | Cleaner synthesis without droplet artifacts |
| GigaGAN | 2023 | Large-scale text-to-image GAN |
StyleGAN maps z into an intermediate latent space W, then injects styles at multiple resolutions. AdaIN and progressive growing helped separate visual factors and produce highly photorealistic faces.
Evaluating GANs
- FID: distance between real and generated feature distributions; lower is better.
- Inception Score: combines confident predictions with class diversity; higher is better.
- Precision and recall: separate sample fidelity from distribution coverage.
- LPIPS: estimates perceptual similarity in a learned feature space.
These metrics are proxies. They can miss semantic errors, memorization, dataset bias, or qualities that matter to human evaluators.
Limitations
- Training instability: The two-player optimization can oscillate or diverge.
- Mode collapse: The generator may produce a narrow set of outputs that reliably fool the discriminator.
- Vanishing gradients: An overconfident discriminator can stop providing useful feedback.
- Artifacts: Transposed convolution can create checkerboard patterns; resize-convolution is a common alternative.
- No explicit likelihood: Standard GANs cannot directly score the probability of a sample.
- Sensitive balance: Architecture, update frequency, learning rates, and regularization often require careful tuning.
- Imperfect evaluation: FID and related metrics do not fully represent quality or diversity.
- Competitive quality: Diffusion models now tend to provide better mode coverage and state-of-the-art synthesis, though usually with slower sampling.
Minibatch discrimination and unrolled GANs can reduce mode collapse. Non-saturating loss, WGAN objectives, spectral normalization, and gradient penalties can improve gradient quality and stability.
Related
- Diffusion Models
- CNN
- Transformer
- Loss Function
- Batch Normalization
- Diffusion Models (2020+) — surpassed GANs in image quality and diversity
- DALL-E (2021) — text-to-image using a transformer and VQ-VAE
- Stable Diffusion (2022) — latent diffusion
- GigaGAN (2023) — GAN-based text-to-image at scale
Sources
- Generative Adversarial Nets — original GAN paper by Goodfellow et al. (NeurIPS 2014)
- Unsupervised Representation Learning with Deep Convolutional GANs — DCGAN by Radford et al. (2015)
- Wasserstein GAN — Arjovsky et al. (2017)
- A Style-Based Generator Architecture for Generative Adversarial Networks — StyleGAN by Karras et al. (CVPR 2019)
- GAN — Wikipedia
- The GAN Zoo — comprehensive list of GAN variants