GAN (Generative Adversarial Network)

What a GAN Does

A generative adversarial network learns to create new samples that resemble its training data. GANs have generated images, video, audio, and 3D content, often in a single fast inference pass.

Introduced in 2014 in Generative Adversarial Nets, the method replaced a hand-designed similarity loss with an adaptive opponent. It dominated image synthesis from roughly 2015 to 2021.

DCGAN (2015), Pix2Pix (2016), CycleGAN (2017), StyleGAN (2018), BigGAN (2018), and StyleGAN2 (2020) progressively improved stability, control, scale, and visual fidelity.

Core Idea

A GAN trains two neural networks in competition:

  • The generator G maps random noise z to a synthetic sample G(z).
  • The discriminator D estimates whether a sample came from the real data rather than the generator.
  • The generator improves by learning which changes make the discriminator more likely to accept its output.

The familiar analogy is a counterfeiter and a detective: each becomes better in response to the other. At the ideal equilibrium, generated and real distributions match, and the discriminator outputs 0.5 everywhere.

GAN Architecture

GAN: Generator vs. Discriminator adversarial training

Source: Goodfellow et al., 2014 — Generative Adversarial Nets (NeurIPS)

How It Works

1. Generate a Candidate

Sample a latent vector z, commonly from a Gaussian distribution. The generator transforms it through learned layers into an output with the same structure as the training data.

A typical DCGAN generator expands a 100-dimensional vector into a 64 × 64 × 3 image:

Layer Operation Output
Input Noise vector z 100
Projection Linear + reshape 4 × 4 × 1024
Block 1 ConvTranspose 4×4/2 + BN + ReLU 8 × 8 × 512
Block 2 ConvTranspose 4×4/2 + BN + ReLU 16 × 16 × 256
Block 3 ConvTranspose 4×4/2 + BN + ReLU 32 × 32 × 128
Output ConvTranspose 4×4/2 + Tanh 64 × 64 × 3

2. Judge Real and Generated Samples

The discriminator is usually a downsampling CNN. It processes either a real or generated image and returns one real/fake score.

Layer Operation Output
Input Real or generated image 64 × 64 × 3
Block 1 Conv 4×4/2 + LeakyReLU 32 × 32 × 128
Block 2 Conv 4×4/2 + BN + LeakyReLU 16 × 16 × 256
Block 3 Conv 4×4/2 + BN + LeakyReLU 8 × 8 × 512
Block 4 Conv 4×4/2 + BN + LeakyReLU 4 × 4 × 1024
Output Flatten + linear + Sigmoid 1

LeakyReLU helps prevent dead discriminator units. Batch normalization stabilizes both networks, except at the discriminator input and generator output.

3. Optimize the Minimax Objective

The original value function is:

min_G max_D V(D, G) = E[log D(x)] + E[log(1 - D(G(z)))]

The discriminator maximizes correct decisions on both real x and generated G(z). The generator minimizes rejection; in practice, the non-saturating objective maximizes log D(G(z)) to provide stronger gradients.

Training alternates discriminator and generator updates. Neither objective is stationary because every update changes the opponent that defines the other network’s learning signal.

4. Condition or Modify the Game

A conditional GAN gives both networks a signal y, such as a class, text prompt, or source image. The pair G(z, y) and D(x, y) enables controlled generation and image-to-image translation.

Wasserstein GAN replaces the original Jensen–Shannon-based game with Earth Mover distance. Weight clipping or a gradient penalty enforces the required Lipschitz constraint and makes the training signal more useful.

Why It Works

  • A learned loss: The discriminator discovers discrepancies that a fixed pixel loss may miss.
  • Implicit distribution learning: The generator need not define or evaluate an explicit probability density.
  • Perceptual sharpness: Adversarial feedback rewards realistic structure instead of averaging plausible outputs into blur.
  • Competitive refinement: Better generated samples force a stronger discriminator, which can provide finer feedback in return.
  • Fast sampling: Once trained, generation is one forward pass rather than an iterative procedure.

This adversarial principle influenced perceptual losses, image restoration, super-resolution, representation learning, self-supervised learning, and adversarial robustness research.

Important Variants

Model Year Main contribution
GAN 2014 Original adversarial framework
cGAN 2014 Generation conditioned on labels or other inputs
DCGAN 2015 Convolutional baseline with batch normalization
Pix2Pix 2016 Paired image-to-image translation
CycleGAN 2017 Unpaired translation through cycle consistency
WGAN 2017 Wasserstein distance for more stable gradients
StyleGAN 2018 Style-based generator and intermediate latent space
BigGAN 2018 Large-scale class-conditional generation
StyleGAN2 2020 Cleaner synthesis without droplet artifacts
GigaGAN 2023 Large-scale text-to-image GAN

StyleGAN maps z into an intermediate latent space W, then injects styles at multiple resolutions. AdaIN and progressive growing helped separate visual factors and produce highly photorealistic faces.

Evaluating GANs

  • FID: distance between real and generated feature distributions; lower is better.
  • Inception Score: combines confident predictions with class diversity; higher is better.
  • Precision and recall: separate sample fidelity from distribution coverage.
  • LPIPS: estimates perceptual similarity in a learned feature space.

These metrics are proxies. They can miss semantic errors, memorization, dataset bias, or qualities that matter to human evaluators.

Limitations

  • Training instability: The two-player optimization can oscillate or diverge.
  • Mode collapse: The generator may produce a narrow set of outputs that reliably fool the discriminator.
  • Vanishing gradients: An overconfident discriminator can stop providing useful feedback.
  • Artifacts: Transposed convolution can create checkerboard patterns; resize-convolution is a common alternative.
  • No explicit likelihood: Standard GANs cannot directly score the probability of a sample.
  • Sensitive balance: Architecture, update frequency, learning rates, and regularization often require careful tuning.
  • Imperfect evaluation: FID and related metrics do not fully represent quality or diversity.
  • Competitive quality: Diffusion models now tend to provide better mode coverage and state-of-the-art synthesis, though usually with slower sampling.

Minibatch discrimination and unrolled GANs can reduce mode collapse. Non-saturating loss, WGAN objectives, spectral normalization, and gradient penalties can improve gradient quality and stability.

Related

Sources