Advanced

22 min read

#Diffusion Models#DDPM#Forward Diffusion#Reverse Diffusion#Noise Schedule#Generative AI

23. Diffusion Models & Generative Denoising (DDPM)

Mathematical foundations and implementations of Denoising Diffusion Probabilistic Models (DDPM): forward Gaussian noise scheduling, reverse denoising U-Net, Score-based models, and latent diffusion.

Diffusion Models: Complete Notes (Beginner to Advanced)

1. Diffusion Models#

Diffusion models are generative models that learn to create data by gradually removing noise from a noisy sample.

The core idea is:

Architecture & Data Flow
Start with real data
      |
      v
Gradually add noise
      |
      v
Almost pure noise
      |
      v
Learn to reverse the process
      |
      v
Start from noise
      |
      v
Gradually denoise
      |
      v
Generated data

Unlike GANs, diffusion models do not require a Generator and Discriminator competing against each other.

A typical diffusion model learns a denoising process that transforms random noise into a meaningful sample.

Basic Generation Flow#

Architecture & Data Flow
Random Noise
     |
     v
Denoising Step
     |
     v
Less Noise
     |
     v
Denoising Step
     |
     v
Less Noise
     |
    ...
     |
     v
Clean Generated Sample

Diffusion models are widely used for:

Image generation
Text-to-image generation
Image editing
Image inpainting
Super-resolution
Audio generation
Other generative tasks

2. Forward Diffusion#

The forward diffusion process gradually adds Gaussian noise to a clean data sample until it becomes approximately pure noise.

Start with:

Mathematical Formulation
x_0 = clean data

After several steps:

›x_0 -> x_1 -> x_2 -> ... -> x_T

where:

Mathematical Formulation
x_T ≈ pure Gaussian noise

Conceptual Flow#

Architecture & Data Flow
Clean Image
    |
 small noise
    v
Slightly noisy image
    |
 more noise
    v
Very noisy image
    |
 more noise
    v
Almost pure noise

The forward process is generally fixed, meaning the noise schedule is chosen rather than learned by the denoising neural network.

3. Forward Diffusion Equation#

A common DDPM formulation defines:

$q(x_t \mid x_{t-1}) = \mathcal{N}\left(x_t;\, \sqrt{1 - \beta_t}\, x_{t-1},\, \beta_t \mathbf{I}\right)$

where:

t = diffusion timestep
beta_t = noise variance schedule at timestep t
I = identity matrix

Define:

$\alpha_t = 1 - \beta_t$

and:

$\bar{\alpha}_t = \prod_{i=1}^t \alpha_i = \alpha_1 \cdot \alpha_2 \dots \alpha_t$

Then x_t can be sampled directly from x_0:

$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$

where:

$\epsilon \sim \mathcal{N}(0, \mathbf{I})$

This equation is extremely useful because we do not need to simulate every previous step to create a noisy version at timestep t.

4. Noise Schedule#

A noise schedule determines how much noise is added at each diffusion timestep.

The schedule defines:

›beta_1, beta_2, ..., beta_T

where beta_t controls the amount of noise introduced at step t.

Conceptually:

Architecture & Data Flow
Small beta
   |
   v
Small amount of noise
   |
   v
Larger beta values
   |
   v
More accumulated noise

Example#

A simple schedule could gradually increase:

›beta_1 < beta_2 < beta_3 < ... < beta_T

The exact schedule is a design choice.

Common schedules used in diffusion research include:

Linear schedules
Cosine schedules
Other improved schedules

Why the Schedule Matters#

If noise is added too slowly or too quickly, the learning problem can become less effective.

The schedule controls the relationship between:

text
signal retained
        and
noise added

at every timestep.

5. Reverse Diffusion#

The reverse diffusion process is the generative process.

The model starts from random noise:

Mathematical Formulation
x_T ~ N(0, I)

and gradually removes noise:

›x_T -> x_{T-1} -> ... -> x_1 -> x_0

Conceptually:

Architecture & Data Flow
Pure Noise
    |
Denoise
    v
Noisy structure
    |
Denoise
    v
More structure
    |
Denoise
    v
Clear structure
    |
Denoise
    v
Generated sample

The reverse process is learned.

Forward vs Reverse#

Architecture & Data Flow
FORWARD

x_0 -> x_1 -> x_2 -> ... -> x_T
       add noise

REVERSE

x_T -> x_{T-1} -> ... -> x_1 -> x_0
       remove noise

The forward process tells us how data becomes noise.

The neural network learns how to approximately reverse that process.

6. Reverse Diffusion Model#

In a DDPM-style model, the reverse transition is modeled as:

›p_theta(x_{t-1} | x_t)

The neural network receives a noisy sample and the timestep:

›(x_t, t)

and predicts information needed to move toward a less noisy sample.

A common parameterization predicts the noise:

›epsilon_theta(x_t, t)

The reverse process can then use this prediction to estimate the previous state.

Basic Architecture#

Architecture & Data Flow
Noisy sample x_t
       +
Timestep t
       |
       v
Denoising Network
       |
       v
Predicted noise
       |
       v
Reverse diffusion step
       |
       v
x_{t-1}

This process is repeated until the final generated sample is obtained.

7. Noise Prediction#

One of the most common diffusion-training approaches is noise prediction.

During training, we start with clean data:

›x_0

sample a random timestep:

›t

and random Gaussian noise:

$\epsilon \sim \mathcal{N}(0, \mathbf{I})$

Then create:

$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$

The model receives:

›x_t, t

and tries to predict the noise that was added:

›epsilon_theta(x_t, t)

Training Objective#

A common objective is:

$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon}\left[\left\| \epsilon - \epsilon_\theta(x_t, t) \right\|^2\right]$

In simple terms:

Architecture & Data Flow
Actual noise
     |
     | compare
     v
Predicted noise

The model learns to estimate the noise component inside the noisy sample.

8. Why Predict Noise?#

At first, it may seem easier to predict the original clean image directly.

However, predicting the noise gives the model a consistent learning target across different timesteps.

For example:

Mathematical Formulation
x_1 = mostly clean + small noise
x_500 = partly noisy
x_T = mostly noise

At each timestep, the model learns:

›"What noise is present in this sample?"

Once the noise is estimated, the reverse process can use that information to move toward a cleaner sample.

Conceptual Flow#

Architecture & Data Flow
Noisy image x_t
       |
       v
Neural Network
       |
       v
Predicted noise epsilon_hat
       |
       v
Remove estimated noise
       |
       v
Less noisy image

9. Denoising#

Denoising is the process of removing noise from a sample.

During generation, the model repeatedly performs denoising steps.

text
x_T
 |
Denoise
 |
x_{T-1}
 |
Denoise
 |
x_{T-2}
 |
...
 |
x_1
 |
Denoise
 |
x_0

The important point is that a diffusion model does not usually transform pure noise into a final image in one neural-network pass.

Instead, generation involves a sequence of denoising steps.

One Denoising Step#

Architecture & Data Flow
Current noisy sample
        |
        +---- timestep information
        |
        v
   Denoising model
        |
        v
Predicted noise / update
        |
        v
Previous, cleaner sample

The exact update depends on the diffusion formulation and sampler.

10. U-Net Diffusion Architecture#

A U-Net is a common neural network architecture used as the denoising model in diffusion systems, especially image diffusion models.

It has an encoder-like downsampling path and a decoder-like upsampling path, connected by skip connections.

Architecture & Data Flow
Input noisy image
       |
       v
   Downsampling
       |
       v
   Downsampling
       |
       v
     Bottleneck
       |
       v
    Upsampling
       |
       v
    Upsampling
       |
       v
Predicted noise

U-Net Structure#

Architecture & Data Flow
                 U-NET

Input
  |
  v
[Down Block] -----------+
  |                     |
  v                     |
[Down Block] --------+  |
  |                  |  |
  v                  |  |
[Bottleneck]         |  |
  |                  |  |
  v                  |  |
[Up Block] <---------+  |
  |                     |
  v                     |
[Up Block] <------------+
  |
  v
Output

The skip connections transfer information from the downsampling path to corresponding upsampling stages.

This helps preserve spatial details.

11. Diffusion U-Net and Timestep Conditioning#

The denoising network needs to know which diffusion timestep it is processing.

The same noisy-looking input can require different operations depending on t.

Therefore, the timestep is provided to the network.

Architecture & Data Flow
Noisy Image x_t
       |
       +--------+
                |
Timestep t -----+
                |
                v
           U-Net
                |
                v
        Predicted Noise

A common approach is to transform the timestep into a learned or fixed embedding and inject that information into intermediate network layers.

Conceptually:

Architecture & Data Flow
t
 |
Timestep Embedding
 |
 +------> U-Net blocks

This allows the same network to perform denoising at different noise levels.

12. Latent Diffusion#

Latent diffusion performs the diffusion process in a learned latent space rather than directly in the original high-dimensional data space.

For images:

Architecture & Data Flow
Image
  |
  v
Encoder
  |
  v
Latent representation
  |
  v
Diffusion process
  |
  v
Denoised latent
  |
  v
Decoder
  |
  v
Image

Instead of adding and removing noise directly in pixel space:

›Pixel space:
Image -> Diffusion -> Image

latent diffusion uses:

Architecture & Data Flow
Latent space:
Image
  -> Encoder
  -> Latent
  -> Diffusion
  -> Latent
  -> Decoder
  -> Image

Why Use Latent Space?#

Images contain a large number of pixel values.

A learned latent representation can be significantly more compact.

Therefore, diffusion in latent space can reduce:

Memory usage
Computational cost
Amount of computation required at high spatial resolution

while still allowing the decoder to reconstruct detailed images.

13. Latent Diffusion Architecture#

A typical latent diffusion system contains:

Architecture & Data Flow
             Input Image
                  |
                  v
            Image Encoder
                  |
                  v
            Latent Representation
                  |
                  v
          +----------------+
          | Diffusion U-Net|
          +----------------+
                  |
                  v
           Denoised Latent
                  |
                  v
            Image Decoder
                  |
                  v
             Output Image

The autoencoder and diffusion model therefore have different responsibilities:

Architecture & Data Flow
Autoencoder:
Image <-> Latent representation

Diffusion model:
Learn how to generate / denoise latent representations

This separation is a key idea behind latent diffusion systems.

14. Text-to-Image Diffusion#

Text-to-image diffusion generates an image based on a text prompt.

Example:

›"An astronaut riding a bicycle on Mars"

The system converts the text into a representation that conditions the diffusion model.

Conceptually:

Architecture & Data Flow
Text Prompt
     |
     v
Text Encoder
     |
     v
Text Embeddings
     |
     v
Conditioned Diffusion Model
     |
     v
Generated Image

In latent diffusion systems, the complete process is approximately:

text
Text
 |
Text Encoder
 |
Text Representation
 |
Random Latent Noise
 |
Diffusion U-Net
 |       ^
 |       |
 |    Text condition
 |
Denoised Latent
 |
Image Decoder
 |
Generated Image

15. Text Conditioning and Cross-Attention#

A common mechanism for injecting text information into an image diffusion model is cross-attention.

The diffusion U-Net contains image/latent features, while the text encoder provides text representations.

Conceptually:

Architecture & Data Flow
Image Latent Features
        |
        | Queries
        v
   Cross-Attention
        ^
        |
   Text Features
    Keys + Values

This allows the denoising network to use information from the text prompt while generating the image.

For example:

›Prompt:
"red car on a snowy mountain"

The text representation provides conditioning information related to:

text
red
car
snowy
mountain

The diffusion model uses this conditioning while progressively denoising the latent representation.

16. Text-to-Image Generation Flow#

A simplified generation pipeline is:

Architecture & Data Flow
Step 1:
Text prompt

        |
        v

Step 2:
Text encoder creates text representation

        |
        v

Step 3:
Start from random noise

        |
        v

Step 4:
Diffusion U-Net predicts noise
using the text condition

        |
        v

Step 5:
Remove predicted noise

        |
        v

Step 6:
Repeat for multiple timesteps

        |
        v

Step 7:
Obtain denoised latent

        |
        v

Step 8:
Decode latent into image

So:

Architecture & Data Flow
Text
  +
Random Noise
  |
  v
Conditioned Denoising
  |
  v
Latent Representation
  |
  v
Image Decoder
  |
  v
Image

17. Forward vs Reverse Diffusion#

Property	Forward Diffusion	Reverse Diffusion
Direction	`x_0 -> x_T`	`x_T -> x_0`
Purpose	Add noise	Remove noise
Process	Usually fixed	Learned
Input	Real data	Noise / noisy sample
Output	Noise	Generated data
Neural network required?	Not necessarily	Yes

Mental Model#

Architecture & Data Flow
FORWARD

Clean data
    |
 Add noise
    |
    v
Pure noise


REVERSE

Pure noise
    |
Remove noise
    |
    v
Generated data

18. Diffusion Models vs GANs#

Both can generate realistic data, but their generation mechanisms are different.

GAN#

Architecture & Data Flow
Random noise
     |
     v
Generator
     |
     v
Sample

Generation can be performed in a single Generator forward pass.

Diffusion#

Architecture & Data Flow
Random noise
     |
     v
Denoising step
     |
     v
Denoising step
     |
    ...
     |
     v
Sample

Generation normally requires multiple denoising steps.

Comparison#

Feature	GAN	Diffusion
Main components	Generator + Discriminator	Denoising model + diffusion process
Training	Adversarial	Noise-prediction / diffusion objective
Generation	Usually one Generator pass	Multiple denoising steps
Common image architecture	CNN / variants	U-Net or related denoiser
Major issue	Mode collapse / unstable training	Sampling can be computationally expensive
Conditioning	Possible	Common and powerful

19. Important Diffusion Terminology#

Diffusion Process#

The overall process of gradually adding and removing noise.

Forward Diffusion#

›Data -> Noise

Reverse Diffusion#

›Noise -> Data

Noise Schedule#

Defines how much noise is introduced at each timestep.

Noise Prediction#

The denoising model predicts the noise component present in a noisy sample.

Denoising#

Removing estimated noise during reverse diffusion.

U-Net#

A common denoising architecture for image diffusion.

Latent Diffusion#

Performs diffusion in a learned latent space rather than directly in pixel space.

Text-to-Image Diffusion#

Uses text conditioning to guide image generation.

20. Summary#

Concept	Simple meaning
Diffusion Model	Generative model that learns to reverse a noise-adding process
Forward Diffusion	Gradually adds noise to data
Reverse Diffusion	Gradually removes noise to generate data
Noise Schedule	Controls noise amount at each timestep
Noise Prediction	Model predicts the noise added to a noisy sample
Denoising	Removes predicted noise during generation
U-Net Diffusion Architecture	Common encoder-decoder denoising network with skip connections
Latent Diffusion	Performs diffusion in a compact latent representation
Text-to-Image Diffusion	Generates images conditioned on text

21. Quick Recap#

Architecture & Data Flow
DIFFUSION

Real Data
   |
   | Forward diffusion
   | add noise
   v
Pure Noise
   |
   | Reverse diffusion
   | remove noise
   v
Generated Data

Architecture & Data Flow
TRAINING

Clean x_0
   |
Choose timestep t
   |
Add known noise epsilon
   |
Create x_t
   |
U-Net(x_t, t)
   |
Predict epsilon
   |
Compare predicted vs actual noise
   |
Loss

Architecture & Data Flow
GENERATION

Random Noise
   |
U-Net + timestep
   |
Denoise
   |
Repeat
   |
Denoised Latent / Image
   |
Decoder if using latent diffusion
   |
Generated Image

Architecture & Data Flow
TEXT-TO-IMAGE

Text Prompt
    |
Text Encoder
    |
Text Representation
    |
    +------------------+
                       |
Random Noise -> Diffusion U-Net
                       |
              Repeated Denoising
                       |
                       v
                 Denoised Latent
                       |
                       v
                 Image Decoder
                       |
                       v
                  Generated Image

One-Line Mental Model#

Mathematical Formulation
Diffusion = Start with noise and learn how to gradually turn that noise into meaningful data.

Knowledge Checkpoint

23. Diffusion Models Checkpoint

Q1.What is the primary mechanism of the Forward Diffusion process in DDPM?

AGradually corrupting clean data x_0 into isotropic Gaussian noise by iteratively adding small amounts of noise according to a variance schedule beta_1...beta_T.

BCompressing images using JPEG discrete cosine transforms.

CUpscaling low-resolution images using bicubic interpolation.

DApplying a discriminator network to reject noisy images.

Q2.What does the neural network (typically a U-Net with time embeddings) predict during Reverse Diffusion in standard DDPM?

AThe exact noise vector epsilon added to data at time step t, which is then subtracted to reconstruct x_{t-1}.

BThe label category of the image.

CThe learning rate schedule for the optimizer.

DThe Fourier spectrum of the image.

Q3.How do Latent Diffusion Models (Stable Diffusion) dramatically accelerate training and inference compared to pixel-space diffusion?

AThey perform the forward and reverse diffusion processes in a compressed, low-dimensional latent space produced by a pretrained VAE encoder/decoder.

BThey skip the reverse process completely.

CThey use single-step linear regression.

DThey only generate 8x8 pixel thumbnails.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Next Lesson

24. Transfer Learning & Domain Adaptation