NeuralDocs

Types of Neural Networks / Generative Adversarial Networks

Generative Adversarial Networks

Advanced

Two networks — a generator and a discriminator — compete, producing highly realistic synthetic data.

The minimax game

The generator G maps random noise z to fake samples; the discriminator D tries to tell real from fake.

Noise zz ~ N(0, I)Generator Gfake G(z)Real datax ~ p_dataDiscrim. DD(·) ∈ [0, 1]Real / Fakeloss → G, D
The generator G turns random noise z into fake samples; the discriminator D sees real and fake samples and outputs the probability that its input is real. D learns to tell them apart while G learns to fool D.
min⁡Gmax⁡D  V(D,G)=Ex∼pdata[log⁡D(x)]+Ez∼pz[log⁡(1−D(G(z)))]\boldsymbol{\begin{aligned}\min_G\max_D\;V(D,G)&=\mathbb{E}_{x\sim p_{\mathrm{data}}}[\log D(x)]\\&\quad+\mathbb{E}_{z\sim p_z}[\log(1-D(G(z)))]\end{aligned}}

The original GAN minimax objective averages real samples x from the data distribution and noise z from a prior p_z. In practice the generator often minimizes −E[log D(G(z))], the non-saturating alternative.

Challenges

  • Mode collapse
  • Training instability
  • Hard-to-measure quality (FID, IS metrics)

Notable variants

  • DCGAN
  • WGAN (Wasserstein loss)
  • CycleGAN (unpaired translation)
  • StyleGAN (high-res faces)

Interview questions & answers

  1. Q1. What are the roles of the generator and discriminator?

    Answer: The generator transforms noise into synthetic samples. The discriminator learns to distinguish real samples from generated ones. Alternating updates train the discriminator to detect fakes and the generator to produce samples that the discriminator judges as real.

  2. Q2. What is mode collapse in GANs?

    Answer: Mode collapse occurs when the generator produces a narrow set of outputs and fails to represent the diversity of the real distribution. A generator can produce plausible individual samples while missing many modes. Alternative objectives and training strategies can help, but do not guarantee a cure.

  3. Q3. Why use the non-saturating generator loss?

    Answer: Early in training, a strong discriminator can give the original minimax generator objective weak gradients. The non-saturating alternative encourages the discriminator to assign high realness to generated samples and often provides a stronger learning signal. It changes the generator objective, not the discriminator's role.

← Previous

Autoencoders & VAEs

Next →

Transformers