Advanced

20 min read

#Autoencoders#VAE#Latent Space#Reparameterization Trick#Dimensionality Reduction#Generative

21. Autoencoders & Variational Autoencoders (VAE)

Unsupervised representation learning: Undercomplete, sparse, and denoising autoencoders, Variational Autoencoder (VAE) probabilistic latent spaces, and KL divergence.

Autoencoders: Complete Notes (Beginner to Advanced)

1. Autoencoder#

An autoencoder is a neural network designed to learn a compact or useful representation of input data and then reconstruct the original input from that representation.

It has two main parts:

Architecture & Data Flow
Input
  |
  v
Encoder
  |
  v
Latent Space
  |
  v
Decoder
  |
  v
Reconstructed Output

The model is trained so that:

Mathematical Formulation
Reconstructed Output ≈ Original Input

Unlike ordinary supervised learning, an autoencoder can learn from the input itself without requiring a separate target label.

Basic Objective#

If the input is:

›x

the autoencoder produces:

Mathematical Formulation
x_hat = Decoder(Encoder(x))

and tries to minimize the difference between:

›x and x_hat

An autoencoder does not automatically learn a useful representation simply because it reconstructs data. The architecture, bottleneck, regularization, noise, and training objective determine what information is learned.

2. Encoder#

The encoder maps the original input into a lower-dimensional or otherwise constrained representation.

Architecture & Data Flow
Input x
   |
   v
Encoder
   |
   v
Latent representation z

Mathematically:

Mathematical Formulation
z = f_encoder(x)

where:

x = input
z = latent representation
f_encoder = encoder function

Example#

Suppose an image is flattened into:

›784 values

The encoder could transform it into:

›784 -> 256 -> 64

So:

Mathematical Formulation
Input dimension = 784
Latent dimension = 64

The encoder must learn which information is useful enough to preserve in the latent representation.

3. Latent Space#

The latent space is the space containing the representations produced by the encoder.

For an input:

›x

the encoder produces:

Mathematical Formulation
z = Encoder(x)

where z is a point in latent space.

Architecture & Data Flow
Input
  |
Encoder
  |
  v
z = latent vector
  |
  v
Latent Space

Example#

Suppose:

Mathematical Formulation
z = [0.4, -0.8, 1.2]

This point represents the input in a three-dimensional latent space.

Real autoencoders may use much larger latent dimensions.

Why a Latent Space Is Useful#

A well-structured latent representation can capture important patterns in the input.

For example, an image latent space might organize information related to:

text
shape
texture
orientation
style
other learned features

However, individual latent dimensions do not necessarily correspond to clean human-interpretable concepts.

4. Decoder#

The decoder maps the latent representation back toward the original input space.

Architecture & Data Flow
Latent vector z
      |
      v
   Decoder
      |
      v
Reconstructed input x_hat

Mathematically:

Mathematical Formulation
x_hat = f_decoder(z)

where:

z = latent representation
x_hat = reconstructed input
f_decoder = decoder function

Example#

If the encoder performs:

›784 -> 256 -> 64

the decoder might perform:

›64 -> 256 -> 784

The decoder learns how to reconstruct the original input from the information stored in the latent representation.

5. Complete Autoencoder Flow#

The complete process is:

Architecture & Data Flow
                AUTOENCODER
                    |
                    v
Input x
  |
  v
+-----------+
|  Encoder  |
+-----------+
      |
      v
Latent z
      |
      v
+-----------+
|  Decoder  |
+-----------+
      |
      v
Reconstruction x_hat
      |
      v
Compare x with x_hat
      |
      v
Reconstruction Loss

During training:

text
x
 |
Encoder
 |
z
 |
Decoder
 |
x_hat
 |
Compare x and x_hat
 |
Loss
 |
Backpropagation
 |
Update parameters

The encoder and decoder are trained together.

6. Reconstruction Loss#

Reconstruction loss measures how different the reconstructed output is from the original input.

The model tries to minimize:

›L(x, x_hat)

where:

Mathematical Formulation
x     = original input
x_hat = reconstructed input

Mean Squared Error#

For continuous-valued inputs, a common choice is Mean Squared Error (MSE):

Mathematical Formulation
MSE = (1/n) Σ (x_i - x_hat_i)^2

Example:

text
Original:
[1.0, 0.5, 0.0]

Reconstruction:
[0.8, 0.6, 0.1]

The loss measures the reconstruction error across the elements.

Binary Cross-Entropy#

For suitable normalized/binary-valued data and output modeling assumptions, Binary Cross-Entropy (BCE) can also be used:

Mathematical Formulation
BCE =
-(1/n) Σ [
    x_i log(x_hat_i)
    +
    (1-x_i) log(1-x_hat_i)
]

The choice of reconstruction loss should match the nature and scaling/distribution of the data.

7. Why the Bottleneck Matters#

Consider an autoencoder:

›784 -> 512 -> 256 -> 784

If the network has enough capacity and there is no useful constraint, it may learn an almost direct identity mapping:

›Input -> Output

Instead, a bottleneck can force the network to compress information:

Architecture & Data Flow
784 -> 128 -> 32 -> 128 -> 784
                  ^
             bottleneck

The encoder must learn a compact representation that retains information useful for reconstruction.

Important Point#

A smaller latent dimension is one way to constrain an autoencoder, but it is not the only way.

Other constraints include:

Noise
Sparsity
Probabilistic regularization
Architectural restrictions

These ideas lead to variants such as denoising autoencoders, sparse autoencoders, and VAEs.

8. Denoising Autoencoder#

A denoising autoencoder is trained to reconstruct a clean input from a corrupted version of that input.

Instead of:

›clean x -> x_hat

the training process is:

Architecture & Data Flow
clean x
   |
Corruption
   |
   v
noisy x_tilde
   |
   v
Encoder
   |
   v
Latent representation
   |
   v
Decoder
   |
   v
Reconstruction x_hat

The target remains the original clean input:

Mathematical Formulation
Target = x

Example#

Clean image:

›[original image]

Corrupted image:

›[noisy image]

The model learns:

›noisy image -> clean image

Why It Helps#

The model cannot simply memorize the corrupted input. It must learn patterns that allow it to recover the underlying clean structure.

This can encourage more robust representations.

9. Denoising Process#

Let:

Mathematical Formulation
x = clean input

A corruption process produces:

Mathematical Formulation
x_tilde = Corrupt(x)

The autoencoder learns:

Mathematical Formulation
z = Encoder(x_tilde)

x_hat = Decoder(z)

while minimizing:

›L(x, x_hat)

Notice the important distinction:

Mathematical Formulation
Input to model  = corrupted x
Target          = clean x

Common Corruption Strategies#

Depending on the data, corruption can include:

Adding noise
Randomly masking input values
Randomly removing features
Image corruption
Token masking

The corruption strategy should be appropriate for the problem.

10. Sparse Autoencoder#

A sparse autoencoder encourages the latent representation to contain mostly small or inactive values.

Instead of only minimizing reconstruction loss:

Mathematical Formulation
L = Reconstruction Loss

the objective includes a sparsity penalty:

Mathematical Formulation
L = Reconstruction Loss + Sparsity Penalty

Conceptually:

Architecture & Data Flow
Input
  |
Encoder
  |
  v
Latent representation
  |
Most activations are small/inactive
  |
  v
Decoder
  |
Reconstruction

Why Sparsity?#

Without constraints, many neurons in the latent representation may be active simultaneously.

Sparsity encourages the model to represent each input using a smaller subset of latent features.

This can lead to more selective feature representations.

11. Sparsity Penalty#

One approach is to encourage the average activation of a hidden unit to remain close to a small target value.

Let:

Mathematical Formulation
rho = desired average activation
rho_hat_j = actual average activation of neuron j

A commonly used penalty is based on KL divergence:

Architecture & Data Flow
KL(rho || rho_hat_j)
=
rho log(rho / rho_hat_j)
+
(1-rho) log((1-rho)/(1-rho_hat_j))

The total sparsity penalty can be:

Mathematical Formulation
Sparsity Penalty
=
beta Σ_j KL(rho || rho_hat_j)

where beta controls the strength of the sparsity constraint.

Overall Objective#

Mathematical Formulation
L_total
=
L_reconstruction
+
beta * L_sparsity

Another practical approach is to directly penalize activation magnitudes, such as with an L1-style penalty.

12. Variational Autoencoder (VAE)#

A Variational Autoencoder (VAE) is a generative model that learns a probabilistic latent representation.

A standard autoencoder maps:

›x -> one latent vector z

A VAE instead learns a probability distribution over latent variables.

Architecture & Data Flow
x
 |
Encoder
 |
+-------------------+
| mean mu           |
| log variance      |
+-------------------+
 |
Sampling
 |
z
 |
Decoder
 |
x_hat

Main Idea#

Instead of saying:

Mathematical Formulation
Input x -> z = [specific vector]

the encoder predicts parameters of a distribution:

Mathematical Formulation
q(z|x) = N(mu, sigma^2)

The latent vector is then sampled from this distribution.

13. VAE Encoder#

The VAE encoder does not directly output only one latent vector.

It typically outputs:

›mu
log(sigma^2)

or equivalent parameters from which the standard deviation can be obtained.

Conceptually:

Architecture & Data Flow
Input x
   |
Encoder
   |
   +----> mu
   |
   +----> log variance

These parameters define the approximate posterior distribution:

›q_phi(z|x)

A common assumption is a diagonal Gaussian:

Mathematical Formulation
q_phi(z|x)
=
N(mu, diag(sigma^2))

14. Reparameterization Trick#

A problem occurs when we directly sample:

Mathematical Formulation
z ~ N(mu, sigma^2)

because the random sampling operation is not directly differentiable in the ordinary sense required for backpropagation.

The reparameterization trick rewrites the sampling process as:

Mathematical Formulation
epsilon ~ N(0, I)

z = mu + sigma * epsilon

where:

Mathematical Formulation
mu    = encoder output
sigma = encoder output
epsilon = random noise

The computation can therefore be differentiated with respect to mu and sigma.

Flow#

Architecture & Data Flow
Input x
   |
Encoder
   |
mu, sigma
   |
epsilon ~ N(0,I)
   |
z = mu + sigma * epsilon
   |
Decoder
   |
x_hat

This is one of the key ideas that makes VAE training practical with gradient-based optimization.

15. VAE Decoder#

The decoder takes a sampled latent vector:

›z

and generates a reconstruction:

Mathematical Formulation
x_hat = Decoder(z)

The decoder therefore learns:

›p_theta(x|z)

Conceptually:

text
Latent z
   |
Decoder
   |
Probability distribution / reconstruction
   |
x_hat

The exact decoder output and reconstruction likelihood depend on the data type and modeling assumptions.

16. VAE Loss#

A VAE does not use only reconstruction loss.

Its objective generally contains two components:

Mathematical Formulation
VAE Loss
=
Reconstruction Loss
+
KL Divergence

More formally, the negative ELBO objective can be written as:

Architecture & Data Flow
L
=
E_q(z|x)[-log p(x|z)]
+
KL(q(z|x) || p(z))

where:

p(z) is the chosen prior, commonly N(0, I)
q(z|x) is the encoder's approximate posterior
p(x|z) is the decoder's likelihood

16.1 Reconstruction Term#

The reconstruction term encourages the decoder to reproduce the input.

Architecture & Data Flow
Input x
   |
Encode -> z
   |
Decode -> x_hat
   |
Compare x and x_hat

16.2 KL Divergence Term#

The KL term encourages the learned latent distributions to remain close to the chosen prior.

Commonly:

Mathematical Formulation
p(z) = N(0, I)

This regularizes the latent space.

17. VAE KL Divergence for a Diagonal Gaussian#

For:

Mathematical Formulation
q(z|x) = N(mu, sigma^2)

and:

Mathematical Formulation
p(z) = N(0, I)

the KL divergence has the closed-form expression:

Architecture & Data Flow
KL(q(z|x) || p(z))
=
-1/2 Σ_i
[
1 + log(sigma_i^2) - mu_i^2 - sigma_i^2
]

This term is minimized when the approximate posterior approaches the standard normal prior.

Why This Matters#

The KL term helps produce a more structured latent space.

Instead of arbitrary isolated latent points, the model is encouraged to keep the latent distributions organized around the chosen prior.

18. VAE as a Generative Model#

A major difference between a standard autoencoder and a VAE is that a VAE is designed to support generation from the latent space.

After training, we can sample:

Mathematical Formulation
z ~ N(0, I)

and pass the sample through the decoder:

text
z
 |
Decoder
 |
Generated sample

Conceptually:

Architecture & Data Flow
Random latent vector
        |
        v
      Decoder
        |
        v
Generated data

This works because the KL regularization encourages the latent space to be compatible with the chosen prior.

19. Autoencoder vs Denoising Autoencoder vs Sparse Autoencoder vs VAE#

Model	Main idea	Main constraint/objective	Typical strength
Autoencoder	Reconstruct input	Reconstruction loss	Representation/compression
Denoising AE	Reconstruct clean input from corrupted input	Reconstruction from noisy input	Robust representations / denoising
Sparse AE	Learn sparse latent activations	Reconstruction + sparsity penalty	Selective feature representations
VAE	Learn probabilistic latent variables	Reconstruction + KL regularization	Generative modeling

20. Autoencoder vs VAE#

Although both have an encoder and decoder, they are conceptually different.

Standard Autoencoder#

text
x
 |
Encoder
 |
z
 |
Decoder
 |
x_hat

The encoder deterministically produces a latent representation.

VAE#

text
x
 |
Encoder
 |
mu, sigma
 |
Sampling
 |
z
 |
Decoder
 |
x_hat

The encoder defines a distribution from which the latent representation is sampled.

Main Difference#

text
Autoencoder:
"Compress this input into a representation."

VAE:
"Learn a structured probabilistic latent space
from which data can also be generated."

21. Practical PyTorch Autoencoder#

A simple fully connected autoencoder can be implemented as:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder = nn.Sequential(
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Linear(256, 64)
        )

        self.decoder = nn.Sequential(
            nn.Linear(64, 256),
            nn.ReLU(),
            nn.Linear(256, 784),
            nn.Sigmoid()
        )

    def forward(self, x):
        z = self.encoder(x)
        x_hat = self.decoder(z)
        return x_hat


model = Autoencoder()
criterion = nn.MSELoss()

x = torch.rand(32, 784)

x_hat = model(x)

loss = criterion(x_hat, x)

print(loss.item())

The training target is the input itself:


🐍 PythonInteractive WebAssembly
loss = criterion(model(x), x)

22. Simple VAE Structure in PyTorch#

A simplified VAE can be structured as:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class VAE(nn.Module):
    def __init__(self, input_dim=784, hidden_dim=256, latent_dim=32):
        super().__init__()

        self.encoder = nn.Linear(input_dim, hidden_dim)

        self.mu_layer = nn.Linear(hidden_dim, latent_dim)
        self.logvar_layer = nn.Linear(hidden_dim, latent_dim)

        self.decoder_hidden = nn.Linear(latent_dim, hidden_dim)
        self.decoder_output = nn.Linear(hidden_dim, input_dim)

    def encode(self, x):
        h = torch.relu(self.encoder(x))

        mu = self.mu_layer(h)
        logvar = self.logvar_layer(h)

        return mu, logvar

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)

        return mu + eps * std

    def decode(self, z):
        h = torch.relu(self.decoder_hidden(z))
        return torch.sigmoid(self.decoder_output(h))

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        x_hat = self.decode(z)

        return x_hat, mu, logvar

A common VAE loss can be written as:


🐍 PythonInteractive WebAssembly
def vae_loss(x_hat, x, mu, logvar):
    reconstruction = nn.functional.binary_cross_entropy(
        x_hat,
        x,
        reduction="sum"
    )

    kl = -0.5 * torch.sum(
        1 + logvar - mu.pow(2) - logvar.exp()
    )

    return reconstruction + kl

The exact reconstruction loss should match the data and decoder likelihood assumptions.

23. Important Conceptual Distinctions#

Encoder vs Decoder#

Architecture & Data Flow
Encoder:
Input -> Latent representation

Decoder:
Latent representation -> Reconstruction / generated output

Latent Space#

›The space in which latent representations live.

Reconstruction Loss#

›Measures how well the decoder reconstructs the target.

Denoising Autoencoder#

›Corrupted input -> Clean reconstruction

Sparse Autoencoder#

›Encourages sparse latent activations.

VAE#

›Learns a probabilistic latent distribution
and regularizes it toward a prior.

24. Summary#

Component / Model	Role
Autoencoder	Learns to reconstruct input through a latent representation
Encoder	Converts input into latent representation
Latent Space	Space containing learned latent representations
Decoder	Converts latent representation back toward input space
Reconstruction Loss	Measures reconstruction error
Denoising Autoencoder	Reconstructs clean data from corrupted input
Sparse Autoencoder	Encourages sparse latent activations
VAE	Learns probabilistic latent representations and supports generation

25. Quick Recap#

Architecture & Data Flow
AUTOENCODER

Input
  |
  v
Encoder
  |
  v
Latent Space
  |
  v
Decoder
  |
  v
Reconstruction
  |
  v
Reconstruction Loss

Architecture & Data Flow
DENOISING AE

Clean Input
    |
Corruption
    |
Noisy Input
    |
Encoder -> Latent -> Decoder
    |
Clean Reconstruction

text
SPARSE AE

Input
  |
Encoder
  |
Sparse Latent Representation
  |
Decoder
  |
Reconstruction

Architecture & Data Flow
VAE

Input
  |
Encoder
  |
mu + sigma
  |
Reparameterization
  |
Latent z
  |
Decoder
  |
Reconstruction
  |
Reconstruction Loss + KL Loss

One-Line Mental Model#

text
Autoencoder       = Learn to reconstruct through a latent representation
Denoising AE      = Reconstruct clean data from corrupted data
Sparse AE         = Learn sparse latent features
VAE               = Learn a probabilistic, regularized latent space for generation

Knowledge Checkpoint

21. Autoencoders Checkpoint

Q1.What is the primary architectural bottleneck that forces an Autoencoder to learn meaningful data representations?

AA lower-dimensional latent space bottleneck z (d_latent << d_input) between the encoder and decoder, preventing identity copying.

BUsing a learning rate of 0.

CSetting all decoder weights to negative numbers.

DDropping 99% of input pixels.

Q2.How do Denoising Autoencoders (DAE) train robust representation features?

AThey corrupt the input with stochastic noise x~ = x + noise, but train the network to reconstruct the original clean input x.

BThey delete random weights during inference.

CThey apply audio filtering to images.

DThey train without a loss function.

Q3.What is the 'Reparameterization Trick' in Variational Autoencoders (VAEs)?

ASampling latent vectors as z = mu + sigma ⊙ epsilon where epsilon ~ N(0, I), allowing backpropagation gradients to flow through deterministic parameters mu and sigma.

BReplacing Gaussian distributions with uniform distributions.

CRe-initializing weights every epoch.

DConverting latent dimensions to integers.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

20. High-Dimensional Embeddings & Vector Representations

Next Lesson

22. Generative Adversarial Networks (GAN)