Free GuideAdvanced

16 min read

#GAN#Generator#Discriminator#Minimax Game#Adversarial Training#Wasserstein GAN#Generative

22. Generative Adversarial Networks (GAN)

Adversarial zero-sum game dynamics: Generator vs Discriminator architectures, minimax loss functions, mode collapse mitigation, and Wasserstein GAN with Gradient Penalty.

Generative Adversarial Networks: Complete Notes (Beginner to Advanced)

1. Generative Adversarial Network (GAN)#

A Generative Adversarial Network (GAN) is a generative deep learning framework in which two neural networks, a Generator and a Discriminator, are trained against each other.

The goal is for the Generator to learn how to produce synthetic data that resembles real training data.

Architecture & Data Flow
                    GAN
                     |
          +----------+----------+
          |                     |
          v                     v
      Generator            Discriminator
          |                     ^
          v                     |
     Fake Samples -------------+
                                |
                         Real / Fake

The two networks have different objectives:

text
Generator:
Produce realistic fake samples.

Discriminator:
Distinguish real and generated samples.

2. Generator#

The Generator produces synthetic samples from a random latent vector.

text
z ~ p(z)

x_fake = G(z)

where z is latent noise, G is the Generator, and x_fake is the generated sample.

Architecture & Data Flow
Random Noise z
      |
      v
+-------------+
|  Generator  |
+-------------+
      |
      v
Fake Sample

Different latent vectors can produce different samples:

Architecture & Data Flow
z1 -> Generator -> Sample 1
z2 -> Generator -> Sample 2
z3 -> Generator -> Sample 3

The Generator learns to produce samples that the Discriminator will classify as real.

3. Discriminator#

The Discriminator receives a sample and estimates whether it came from the real training distribution or the Generator.

In the original GAN formulation:

›D(x) ≈ probability that x is real

Therefore, during successful discriminator training:

›D(real) -> close to 1
D(fake) -> close to 0

Architecture & Data Flow
Real Sample --------+
                    |
                    v
                Discriminator
                    |
                    v
                Real / Fake
                    ^
                    |
Generated Sample ---+

The Discriminator provides the learning signal that helps the Generator improve.

4. Adversarial Training#

GANs use adversarial training, where the Generator and Discriminator have competing objectives.

text
Generator:
"Make fake samples look real."

Discriminator:
"Tell real and fake samples apart."

Training Loop#

text
1. Sample real data
2. Sample random latent vector z
3. Generate fake data
4. Train Discriminator on real and fake data
5. Train Generator to fool Discriminator
6. Repeat

Architecture & Data Flow
                 Random z
                     |
                     v
                Generator
                     |
                     v
                  Fake
                     |
       +-------------+-------------+
       |                           |
       v                           v
   Train D                      Train G
       |                           |
 Real -> 1                     Fake -> 1
 Fake -> 0                     through D

The Generator does not directly compare its output with a specific target sample in the standard GAN setup. It learns through the Discriminator's feedback.

5. GAN Loss#

The original GAN formulation uses the minimax objective:

text
min_G max_D V(D,G)
=
E_{x~p_data} [log D(x)]
+
E_{z~p(z)} [log(1-D(G(z)))]

Discriminator Objective#

The Discriminator wants:

›D(x) -> 1

for real samples and:

›D(G(z)) -> 0

for generated samples.

Its objective can be viewed as maximizing:

›log D(x) + log(1 - D(G(z)))

Generator Objective#

In the original minimax formulation, the Generator minimizes:

›log(1 - D(G(z)))

A commonly used non-saturating Generator loss is instead:

›L_G = -E_z [log D(G(z))]

This provides a stronger learning signal when the Discriminator is initially very confident.

Practical BCE Form#

Python
# Discriminator
loss_D = BCE(D(real), 1) + BCE(D(fake.detach()), 0)

# Generator
loss_G = BCE(D(fake), 1)

detach() prevents the Discriminator update from changing Generator parameters during that step.

6. Adversarial Training in Detail#

Step 1: Train the Discriminator#

Sample:

›x ~ p_data
z ~ p(z)

Generate:

›x_fake = G(z)

Then train the Discriminator:

›D(x)      -> target 1
D(x_fake) -> target 0

7. GAN Equilibrium#

Ideally:

›p_g(x) ≈ p_data(x)

where p_g is the Generator's distribution.

At the ideal theoretical equilibrium, the Discriminator cannot distinguish the two distributions better than chance:

›D(x) ≈ 0.5

for samples drawn from the common distribution.

A discriminator output of 0.5 does not literally mean that a particular image is "50% real." It means that, at the ideal equilibrium of the original GAN formulation, the Discriminator has no useful distinguishing ability.

8. Mode Collapse#

Mode collapse occurs when the Generator produces insufficient variety.

Suppose the real dataset contains:

text
Red cars
Blue cars
Green cars
Black cars
White cars

A collapsed Generator might repeatedly produce:

text
Red cars
Red cars
Red cars
Red cars

The samples can look realistic while still lacking diversity.

Normal Behavior#

Architecture & Data Flow
z1 -> different sample
z2 -> different sample
z3 -> different sample
z4 -> different sample

Mode Collapse#

Architecture & Data Flow
z1 -> similar sample
z2 -> similar sample
z3 -> similar sample
z4 -> similar sample

Causes#

GAN optimization is a difficult two-player optimization problem. Contributing factors can include:

Unstable Generator-Discriminator dynamics
The Generator finding a small region that reliably fools the Discriminator
Insufficient diversity in the learning signal
Imbalance between Generator and Discriminator learning

Effect#

text
Real distribution:
many modes

Generated distribution:
few modes

Therefore:

text
High sample quality
+
Low diversity

can occur simultaneously.

9. Conditional GAN#

A Conditional GAN (cGAN) extends a GAN by providing additional information, called a condition, to both networks.

Instead of:

›G(z)

we use:

›G(z, y)

where y is the condition.

For example:

Mathematical Formulation
y = "cat"

can instruct the Generator to create a cat image.

Architecture & Data Flow
Random z + condition y
          |
          v
     Generator
          |
          v
     Fake sample
          |
          v
Discriminator receives
sample + condition
          |
          v
       Score

Example#

For handwritten digits:

›z + label 3 -> Generator -> image of 3
z + label 7 -> Generator -> image of 7

The condition gives control over the type of generated sample.

Conditional Objective#

A conditional version of the GAN objective can be written as:

text
min_G max_D
E_{x,y~p_data}[log D(x,y)]
+
E_{z~p(z),y~p(y)}
[log(1-D(G(z,y),y))]

The exact conditioning mechanism varies across implementations.

10. DCGAN#

DCGAN stands for Deep Convolutional Generative Adversarial Network.

It is a GAN architecture designed for image generation using convolutional neural networks.

Architecture & Data Flow
                 Random Noise
                      |
                      v
                  Generator
                      |
            Learned Upsampling
                      |
                  Fake Image
                      |
                      v
                Discriminator
                      ^
                      |
                  Real Image

DCGAN introduced architectural practices that made convolutional GANs more stable and effective than many earlier designs.

11. DCGAN Generator#

The Generator progressively increases spatial resolution.

A typical conceptual flow is:

Architecture & Data Flow
Latent vector
    |
Projection
    |
4 x 4 feature map
    |
8 x 8
    |
16 x 16
    |
32 x 32
    |
64 x 64 image

The original DCGAN work commonly used transposed convolutions for learned upsampling.

Typical practices included:

Convolutional architecture instead of fully connected hidden layers where practical
Transposed convolutions in the Generator
Batch normalization in many Generator layers
ReLU in Generator hidden layers
tanh at the output for appropriately scaled image data

Exact implementations can vary.

12. DCGAN Discriminator#

The Discriminator progressively reduces spatial dimensions while learning increasingly high-level features.

Architecture & Data Flow
64 x 64 image
      |
Convolution + stride
      |
32 x 32
      |
Convolution + stride
      |
16 x 16
      |
Convolution + stride
      |
8 x 8
      |
...
      |
Real/Fake score

Typical DCGAN practices included:

Strided convolutions for downsampling
Convolutional layers instead of traditional pooling where practical
LeakyReLU in the Discriminator
Batch normalization in many layers
Avoiding fully connected hidden layers where practical

13. DCGAN Architecture Example#

Architecture & Data Flow
                GENERATOR

Random z
   |
Projection
   |
4 x 4 x C
   |
Transposed Conv
   |
8 x 8 x C
   |
Transposed Conv
   |
16 x 16 x C
   |
Transposed Conv
   |
32 x 32 x C
   |
Transposed Conv
   |
64 x 64 x 3
   |
Fake Image

Architecture & Data Flow
                DISCRIMINATOR

64 x 64 x 3 Image
        |
    Conv + Stride
        |
    32 x 32
        |
    Conv + Stride
        |
    16 x 16
        |
    Conv + Stride
        |
     8 x 8
        |
       ...
        |
  Real/Fake Score

14. GAN vs Conditional GAN vs DCGAN#

Model	Main idea	Typical use
GAN	Generator and Discriminator compete	General generative modeling
Conditional GAN	GAN receives additional conditions	Controlled generation
DCGAN	GAN architecture using convolutional networks	Image generation

These ideas can also be combined. For example, a convolutional GAN can additionally be conditioned on class labels.

15. GAN Training vs Autoencoder Training#

GANs and autoencoders are both generative-modeling approaches, but their training mechanisms differ.

Autoencoder#

text
Input
  |
Encoder
  |
Latent
  |
Decoder
  |
Reconstruction
  |
Reconstruction Loss

The reconstruction is directly compared with a target.

GAN#

Architecture & Data Flow
Random Noise
     |
Generator
     |
Fake Sample
     |
Discriminator
     |
Adversarial Loss

The Generator learns through the Discriminator's feedback.

Key Difference#

text
Autoencoder:
"Reconstruct this input."

GAN:
"Generate a sample that fools the Discriminator."

16. Simple GAN Concept in PyTorch#

A simplified fully connected GAN can be implemented as:

Python
import torch
import torch.nn as nn

class Generator(nn.Module):
    def __init__(self, latent_dim=100, output_dim=784):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, output_dim),
            nn.Sigmoid()
        )

    def forward(self, z):
        return self.network(z)


class Discriminator(nn.Module):
    def __init__(self, input_dim=784):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 128),
            nn.LeakyReLU(0.2),
            nn.Linear(128, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.network(x)

This is a simplified GAN rather than a DCGAN because it uses fully connected layers.

17. Conceptual GAN Training Code#

Python
criterion = nn.BCELoss()

optimizer_G = torch.optim.Adam(
    generator.parameters(),
    lr=0.0002
)

optimizer_D = torch.optim.Adam(
    discriminator.parameters(),
    lr=0.0002
)

for real_images, _ in dataloader:

    batch_size = real_images.size(0)

    real_images = real_images.view(batch_size, -1)

    real_labels = torch.ones(batch_size, 1)
    fake_labels = torch.zeros(batch_size, 1)

    # Train Discriminator
    z = torch.randn(batch_size, latent_dim)

    fake_images = generator(z)

    real_pred = discriminator(real_images)
    fake_pred = discriminator(fake_images.detach())

    loss_D_real = criterion(real_pred, real_labels)
    loss_D_fake = criterion(fake_pred, fake_labels)

    loss_D = loss_D_real + loss_D_fake

    optimizer_D.zero_grad()
    loss_D.backward()
    optimizer_D.step()

    # Train Generator
    z = torch.randn(batch_size, latent_dim)

    fake_images = generator(z)
    fake_pred = discriminator(fake_images)

    loss_G = criterion(fake_pred, real_labels)

    optimizer_G.zero_grad()
    loss_G.backward()
    optimizer_G.step()

This demonstrates the basic alternating optimization process.

Production GAN implementations often use more advanced architectures, objectives, regularization, and training techniques.

18. Important GAN Terminology#

Real Sample#

A sample drawn from the real data distribution:

›x ~ p_data

Fake Sample#

A sample generated by:

›x_fake = G(z)

Latent Vector#

Random input given to the Generator:

›z ~ p(z)

Concept	Simple meaning
GAN	Generative framework with competing Generator and Discriminator
Generator	Creates synthetic samples from latent noise
Discriminator	Distinguishes real samples from generated samples
Adversarial Training	Generator and Discriminator learn through competing objectives
GAN Loss	Objective used to train the two networks
Mode Collapse	Generator produces limited varieties of samples
Conditional GAN	GAN controlled using additional conditions
DCGAN	GAN architecture built around convolutional networks

20. Quick Recap#

Architecture & Data Flow
                    GAN
                     |
          +----------+----------+
          |                     |
          v                     v
      Generator            Discriminator
          |                     ^
          v                     |
       Fake Data --------------+
                                |
                           Real / Fake

Architecture & Data Flow
GAN
 -> Generator + Discriminator
 -> Adversarial training
 -> Generate realistic samples

Generator
 -> z -> G(z) -> fake sample

Discriminator
 -> sample -> real/fake score

Mode Collapse
 -> Low diversity in generated samples

Conditional GAN
 -> Generate based on condition

DCGAN
 -> GAN + convolutional architecture
 -> Mainly designed for image generation

One-Line Mental Model#

text
GAN = Generator tries to fool the Discriminator,
      while the Discriminator tries to detect the Generator's fakes.

22. Generative Adversarial Networks (GAN)

Generative Adversarial Networks: Complete Notes (Beginner to Advanced)

Finished studying this notebook?