∇GradientSchool
Intermediate
24 min read
#Sequence Models#RNN#BPTT#Hidden States#Vanishing Gradients#NLP

14. Sequence Models & Recurrent Neural Networks (RNN)

Processing temporal and sequential data: Recurrent Neural Network (RNN) hidden state dynamics, Backpropagation Through Time (BPTT), and vanishing gradients.

Sequence Models: Complete Notes (Beginner to Advanced)

Introduction#

Sequence Models are neural network models designed to process data where the order of elements matters.

In ordinary tabular data, changing the order of rows usually does not change the meaning of an individual row.

In sequential data, order is part of the information.

Examples include:

text
Words in a sentence
Time-series measurements
Stock prices over time
Speech signals
Sensor readings
User activity sequences

A sequence model processes information across multiple time steps while maintaining some representation of previously seen information.

A simplified sequence-model flow is:

text
x₁ → [RNN] → h₁
             ↓
x₂ → [RNN] → h₂
             ↓
x₃ → [RNN] → h₃
             ↓
x₄ → [RNN] → h₄

The important idea is that the computation at the current time step can depend on information from previous time steps.

1. Sequential Data

Sequential Data is data in which the order of observations carries meaning.

Example: Sentence#

Consider:

›"I love machine learning"

The order is:

›I → love → machine → learning

Changing the order can change the meaning:

›"Learning machine love I"

Therefore, a model needs to consider the sequence order.

Example: Time Series#

Suppose a temperature sensor produces:

›25°C → 26°C → 27°C → 29°C

The value at one time step is part of a temporal sequence.

A model may use previous observations to understand the current state or predict a future value.

Sequence Representation#

A sequence can be represented as:

Mathematical Formulation
X = [x₁, x₂, x₃, ..., x_T]

Where:

Mathematical Formulation
xₜ = input at time step t
T  = sequence length

For a sentence:

Mathematical Formulation
x₁ = "I"
x₂ = "love"
x₃ = "machine"
x₄ = "learning"

For a sensor:

Mathematical Formulation
x₁ = reading at time 1
x₂ = reading at time 2
...
x_T = reading at time T

Why Ordinary Feedforward Networks Are Not Ideal#

A basic feedforward network does not naturally maintain information from previous time steps.

Conceptually:

text
x₁ → Network → output₁
x₂ → Network → output₂
x₃ → Network → output₃

The network treats each input independently unless previous information is explicitly provided.

An RNN introduces a hidden state so that information can flow from one time step to the next.

2. Recurrent Neural Networks (RNN)

A Recurrent Neural Network (RNN) is a neural network designed to process sequential data by maintaining a hidden state.

The hidden state acts as a form of memory that carries information from previous time steps.

Basic RNN Idea#

At time step t, the RNN receives:

›Current input: xₜ
Previous hidden state: hₜ₋₁

It produces:

›Current hidden state: hₜ

The basic relationship is:

Mathematical Formulation
hₜ = f(Wₓh xₜ + Wₕh hₜ₋₁ + bₕ)

Where:

Mathematical Formulation
xₜ       = current input
hₜ₋₁     = previous hidden state
hₜ       = current hidden state
Wₓh      = input-to-hidden weights
Wₕh      = hidden-to-hidden weights
bₕ       = hidden bias
f        = activation function

An output can then be calculated as:

Mathematical Formulation
yₜ = g(Wₕy hₜ + bᵧ)

Where:

Mathematical Formulation
Wₕy = hidden-to-output weights
bᵧ = output bias
g = output activation

Core RNN Flow#

Architecture & Data Flow
        hₜ₋₁
          |
          v
xₜ ---> [ RNN Cell ] ---> hₜ ---> yₜ

At the next time step:

Architecture & Data Flow
hₜ
 |
 v
[ RNN Cell ] <--- xₜ₊₁
 |
 v
hₜ₊₁

Thus, information flows through the sequence.

The Important Point: Shared Weights#

The same RNN parameters are reused at every time step.

text
x₁ → [RNN Cell] → h₁
x₂ → [same RNN Cell] → h₂
x₃ → [same RNN Cell] → h₃

The cell is conceptually repeated, but the weights are shared.

This allows the same learned transformation to process sequences of different lengths.

3. Hidden State

The Hidden State is the internal representation maintained by an RNN as it processes a sequence.

It contains information produced from the current input and the previous hidden state.

The update is:

Mathematical Formulation
hₜ = f(Wₓh xₜ + Wₕh hₜ₋₁ + bₕ)

Intuition#

Suppose an RNN reads:

›"The dog chased the ..."

As it processes the sequence, the hidden state can encode information about previously processed words.

Conceptually:

text
"The"
  ↓
h₁

"dog"
  ↓
h₂  ← contains information influenced by "The"

"chased"
  ↓
h₃  ← contains information influenced by earlier words

"the"
  ↓
h₄

...

The hidden state is not a literal copy of previous inputs.

Instead, it is a learned numerical representation.

Hidden State as Memory#

A useful mental model is:

text
Previous information
       ↓
Previous hidden state
       ↓
Current RNN computation
       ↑
Current input
       ↓
New hidden state

Therefore:

Mathematical Formulation
hₜ = memory of relevant sequence information up to time t

This is an intuition rather than a guarantee that every detail from the past is preserved.

Initial Hidden State#

At the beginning of a sequence, there is no previous hidden state.

Commonly:

Mathematical Formulation
h₀ = 0

or the initial state can be learned or provided by another network.

With zero initialization:

text
x₁ + h₀ → h₁
x₂ + h₁ → h₂
x₃ + h₂ → h₃

4. Recurrent Connections

A Recurrent Connection is the connection that carries information from a previous time step to the current time step.

In an RNN:

text
hₜ₋₁ ───────────────→ hₜ
          recurrent
          connection

This creates the recurrence.

Feedforward vs Recurrent#

Feedforward network:

›x → layer → layer → output

Information moves forward through layers.

RNN:

text
x₁ → h₁ → h₂ → h₃ → h₄
       ↑    ↑    ↑
      x₂   x₃   x₄

Information moves through time as well as through the network.

Recurrent Weight Matrix#

The recurrent transformation is controlled by:

›Wₕh

The same matrix is reused at every time step:

text
h₁ → Wₕh → h₂
h₂ → Wₕh → h₃
h₃ → Wₕh → h₄

This parameter sharing is one of the defining characteristics of an RNN.

Why Recurrence Matters#

Without recurrence:

text
x₁ → output₁
x₂ → output₂
x₃ → output₃

With recurrence:

text
x₁ → h₁
      ↓
x₂ → h₂
      ↓
x₃ → h₃

The current state can therefore depend on previous sequence information.

5. Unrolling

Unrolling is the process of representing an RNN across its time steps as a sequence of repeated computational cells.

The RNN is recurrent by definition, but unrolling makes the temporal computation explicit.

Compact RNN Representation#

text
        ┌───────────────┐
xₜ ────>│    RNN Cell   │───> hₜ
        └───────▲───────┘
                |
              hₜ₋₁

Unrolled Representation#

For a sequence of four inputs:

Architecture & Data Flow
x₁        x₂        x₃        x₄
 |         |         |         |
 v         v         v         v
[RNN] →  [RNN] →  [RNN] →  [RNN]
 |         |         |         |
 v         v         v         v
h₁        h₂        h₃        h₄

The same RNN parameters are used at every step.

Unrolled Equations#

Mathematical Formulation
h₁ = f(Wₓh x₁ + Wₕh h₀ + b)

h₂ = f(Wₓh x₂ + Wₕh h₁ + b)

h₃ = f(Wₓh x₃ + Wₕh h₂ + b)

h₄ = f(Wₓh x₄ + Wₕh h₃ + b)

Notice that:

text
h₂ depends on h₁
h₃ depends on h₂
h₄ depends on h₃

Therefore, information can propagate through the sequence.

Why Unrolling Is Important#

Unrolling makes it possible to:

  • Understand how information moves through time
  • Visualize recurrent dependencies
  • Compute gradients through the sequence
  • Understand Backpropagation Through Time (BPTT)

6. Backpropagation Through Time (BPTT)

Backpropagation Through Time (BPTT) is the training procedure used to calculate gradients for an unrolled RNN.

The key idea is simple:

text
Unroll the RNN
      ↓
Treat each time step as part of a computation graph
      ↓
Compute outputs and loss
      ↓
Backpropagate gradients backward through time
      ↓
Update shared parameters

Forward Pass#

Suppose the sequence is:

›x₁ → x₂ → x₃

The forward computation is:

text
x₁ → h₁ → h₂ → h₃
             ↓
            ...

Each hidden state depends on the previous state.

Loss#

Suppose each time step produces an output:

›y₁, y₂, y₃

with corresponding targets:

›target₁, target₂, target₃

The total sequence loss can be represented as:

Mathematical Formulation
L = L₁ + L₂ + L₃

The exact loss depends on the task.

Backward Pass#

Because:

›h₃ depends on h₂
h₂ depends on h₁

the gradient must flow backward through the time steps:

›h₃ ← h₂ ← h₁

Conceptually:

text
Forward:

x₁ → h₁ → h₂ → h₃ → output


Backward:

      gradient
          ←
h₁ ← h₂ ← h₃

Shared Parameters Receive Contributions From Multiple Time Steps#

The same parameters are used repeatedly:

text
Wₓh
Wₕh
Wₕy

Therefore, gradients from different time steps contribute to the update of the same parameters.

Conceptually:

text
Time 1 gradient ──┐
Time 2 gradient ──┼──> shared parameter gradient
Time 3 gradient ──┘

Why BPTT Can Become Difficult#

When sequences are long, gradients must pass through many recurrent steps.

This can lead to:

text
Vanishing gradients
or
Exploding gradients

Vanishing Gradient

The gradient becomes progressively smaller as it propagates backward.

Conceptually:

text
Large gradient
     ↓
smaller
     ↓
smaller
     ↓
almost zero

This makes learning long-range dependencies difficult.

Exploding Gradient

The gradient can become extremely large:

text
Small
  ↓
larger
  ↓
very large
  ↓
unstable

Gradient clipping is a common technique used to limit excessively large gradients.

Truncated BPTT#

For very long sequences, BPTT can be computationally expensive.

Truncated BPTT limits the backward pass to a fixed number of time steps.

For example:

text
Sequence:

x₁ x₂ x₃ x₄ x₅ x₆ x₇ x₈

Backward windows:

[x₁ x₂ x₃]
      [x₄ x₅ x₆]
            [x₇ x₈]

The exact truncation strategy depends on the implementation.

The important idea is:

text
Full BPTT
→ backpropagate through the complete sequence

Truncated BPTT
→ backpropagate through shorter windows

7. Bidirectional RNN

A Bidirectional RNN processes a sequence in both directions:

text
Forward direction:
x₁ → x₂ → x₃ → x₄

Backward direction:
x₄ → x₃ → x₂ → x₁

This allows the representation at a position to use information from both earlier and later elements.

Why Bidirectional Processing Helps#

Consider:

›"I went to the bank to deposit money."

The word "bank" can be interpreted using surrounding context.

A forward-only RNN primarily builds context from earlier elements.

A bidirectional RNN can use:

text
Past context
+
Future context

Architecture#

text
Forward RNN:

x₁ → h₁ᶠ → h₂ᶠ → h₃ᶠ → h₄ᶠ


Backward RNN:

x₁ ← h₁ᵇ ← h₂ᵇ ← h₃ᵇ ← h₄ᵇ

At each position, the forward and backward hidden states are combined.

Commonly:

Mathematical Formulation
hₜ = [hₜᶠ ; hₜᵇ]

where ; denotes concatenation.

Bidirectional Flow#

Architecture & Data Flow
                 x₁
                  |
        +---------+---------+
        |                   |
        v                   v
 Forward RNN           Backward RNN
        |                   |
        v                   v
      h₁ᶠ                 h₁ᵇ
        \                   /
         \                 /
          +---------------+
                  |
                  v
          Combined State

For every time step:

Mathematical Formulation
Combined representation
=
Forward representation
+
Backward representation

More precisely, the two vectors are typically concatenated.

Advantages#

Bidirectional RNNs can capture:

  • Past context
  • Future context
  • More complete contextual representations

They can be useful for tasks such as:

text
Sequence labeling
Named entity recognition
Speech processing
Text classification

Important Limitation#

A bidirectional RNN requires access to the sequence in both directions.

Therefore, it is generally unsuitable for strictly causal settings where the model must make a prediction using only information available up to the current time.

For example:

›Real-time next-value prediction

may require a causal/forward-only model.

8. Stacked RNN

A Stacked RNN contains multiple RNN layers placed on top of one another.

Instead of having one recurrent layer:

text
Input
  ↓
RNN Layer
  ↓
Output

we use:

text
Input
  ↓
RNN Layer 1
  ↓
RNN Layer 2
  ↓
RNN Layer 3
  ↓
Output

Why Stack RNN Layers?#

Different layers can learn different levels of representation.

Conceptually:

text
Layer 1
→ lower-level temporal patterns

Layer 2
→ higher-level temporal patterns

Layer 3
→ more abstract sequence representation

The exact features learned depend on the data and task.

Stacked RNN at One Time Step#

For time step t:

text
xₜ
 ↓
RNN Layer 1
 ↓
hₜ⁽¹⁾
 ↓
RNN Layer 2
 ↓
hₜ⁽²⁾
 ↓
RNN Layer 3
 ↓
hₜ⁽³⁾

The output of one recurrent layer becomes the input to the next recurrent layer.

Stacked RNN Across Time#

text
             Time →
        t₁        t₂        t₃        t₄

Layer 3  h₁³  →  h₂³  →  h₃³  →  h₄³
          ↑         ↑         ↑         ↑
Layer 2  h₁²  →  h₂²  →  h₃²  →  h₄²
          ↑         ↑         ↑         ↑
Layer 1  h₁¹  →  h₂¹  →  h₃¹  →  h₄¹
          ↑         ↑         ↑         ↑
Input     x₁        x₂        x₃        x₄

Each layer has its own parameters.

However, within a particular layer, the recurrent parameters are shared across time steps.

Stacked vs Bidirectional RNN#

These concepts solve different problems.

Stacked RNN:

text
More recurrent layers
↓
Greater depth

Bidirectional RNN:

text
Two temporal directions
↓
Past + future context

They can also be combined:

text
Input
  ↓
Bidirectional RNN Layer 1
  ↓
Bidirectional RNN Layer 2
  ↓
Output

9. RNN Computation Example

Consider a simple RNN with:

Mathematical Formulation
xₜ = current input
hₜ = hidden state

and:

Mathematical Formulation
hₜ = tanh(Wₓh xₜ + Wₕh hₜ₋₁ + b)

Suppose:

Mathematical Formulation
Wₓh = 0.5
Wₕh = 0.8
b = 0
x₁ = 1
h₀ = 0

Then:

Mathematical Formulation
h₁ = tanh(0.5 × 1 + 0.8 × 0)
   = tanh(0.5)

Approximately:

Mathematical Formulation
h₁ ≈ 0.462

Now suppose:

Mathematical Formulation
x₂ = 2

Then:

Mathematical Formulation
h₂ = tanh(0.5 × 2 + 0.8 × 0.462)
Mathematical Formulation
h₂ = tanh(1 + 0.3696)
Mathematical Formulation
h₂ = tanh(1.3696)

Approximately:

Mathematical Formulation
h₂ ≈ 0.879

Notice that h₂ depends on both:

›x₂

and:

›h₁

This demonstrates the core recurrent mechanism.

10. Simple RNN Implementation with NumPy

A minimal RNN cell can be implemented as:

🐍 Python
import numpy as np

# Parameters
Wxh = np.array([[0.5]])
Whh = np.array([[0.8]])
b = np.array([[0.0]])

# Sequence
x = [1.0, 2.0, 3.0]

# Initial hidden state
h = np.array([[0.0]])

for xt in x:
    xt = np.array([[xt]])

    h = np.tanh(Wxh @ xt + Whh @ h + b)

    print("hidden state:", h.item())

The important computation is:

🐍 Python
h = np.tanh(Wxh @ xt + Whh @ h + b)

Here:

›Wxh @ xt

represents the effect of the current input.

›Whh @ h

represents information carried from the previous time step.

11. RNN with PyTorch

PyTorch provides an nn.RNN module.

🐍 Python
import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=10,
    hidden_size=20,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, hidden = rnn(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

Here:

Mathematical Formulation
batch_size = 4
sequence_length = 5
input_size = 10
hidden_size = 20

Therefore:

Mathematical Formulation
x shape
= (4, 5, 10)

The output contains hidden representations for the sequence.

For a single-direction, single-layer RNN:

Mathematical Formulation
output shape
= (4, 5, 20)

The final hidden state has shape:

›(1, 4, 20)

The first dimension represents:

›number of layers × number of directions

12. Bidirectional RNN with PyTorch

A bidirectional RNN can be created with:

🐍 Python
import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=10,
    hidden_size=20,
    num_layers=1,
    batch_first=True,
    bidirectional=True
)

x = torch.randn(4, 5, 10)

output, hidden = rnn(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

Because there are two directions:

›forward
backward

the output feature dimension becomes:

›2 × hidden_size

Therefore:

Mathematical Formulation
output shape
= (4, 5, 40)

The hidden-state first dimension becomes:

Mathematical Formulation
num_layers × num_directions
= 1 × 2
= 2

So:

Mathematical Formulation
hidden shape
= (2, 4, 20)

13. Stacked RNN with PyTorch

Multiple RNN layers can be created using num_layers.

🐍 Python
import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=10,
    hidden_size=20,
    num_layers=3,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, hidden = rnn(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

Here:

Mathematical Formulation
num_layers = 3

so there are three recurrent layers.

For a single-direction RNN:

Mathematical Formulation
output shape
= (4, 5, 20)

hidden shape
= (3, 4, 20)

The hidden-state first dimension is:

›number of layers × number of directions

14. Bidirectional + Stacked RNN

The two ideas can be combined.

🐍 Python
rnn = nn.RNN(
    input_size=10,
    hidden_size=20,
    num_layers=3,
    batch_first=True,
    bidirectional=True
)

There are:

Mathematical Formulation
3 layers
×
2 directions
=
6 recurrent hidden-state groups

The output feature size is:

Mathematical Formulation
2 × 20 = 40

For an input:

Mathematical Formulation
(batch=4, sequence=5, features=10)

the shapes are:

text
Input:
(4, 5, 10)

Output:
(4, 5, 40)

Hidden:
(6, 4, 20)

15. Parameter Sharing Across Time

One of the most important characteristics of an RNN is that the same parameters are reused at every time step.

Suppose the sequence has four steps:

text
x₁ → RNN
x₂ → RNN
x₃ → RNN
x₄ → RNN

It may look like there are four different RNNs, but there are not.

Conceptually:

text
             Same parameters
          ┌───────────────────┐
          ↓                   ↓
x₁ → [RNN] → h₁
x₂ → [RNN] → h₂
x₃ → [RNN] → h₃
x₄ → [RNN] → h₄

The same:

text
Wₓh
Wₕh
bₕ

are reused.

This allows the model to process sequences with varying lengths without creating a completely new set of parameters for every possible sequence length.

16. Sequence-to-Sequence Behavior

RNNs can be used in different input-output arrangements.

One-to-One#

Although not the main purpose of sequence models:

›Input → Output

One-to-Many#

text
Input
 ↓
RNN
 ↓
Sequence Output

Many-to-One#

text
Sequence
 ↓
RNN
 ↓
Single Output

Example:

text
Sentence
   ↓
RNN
   ↓
Sentiment

Many-to-Many#

text
x₁ → h₁ → y₁
x₂ → h₂ → y₂
x₃ → h₃ → y₃

Example:

text
Input sequence
   ↓
Sequence labeling
   ↓
Output label at each time step

The exact input-output structure depends on the application.

17. Important Terminology

TermMeaning
Sequential DataData where order matters
RNNNeural network designed to process sequences
Hidden StateInternal representation carrying information through time
Recurrent ConnectionConnection carrying previous hidden information to the next step
UnrollingExpanding recurrent computation across time steps
BPTTBackpropagation through the unrolled time steps
Bidirectional RNNRNN processing the sequence forward and backward
Stacked RNNMultiple RNN layers arranged vertically

18. Summary

ConceptCore Idea
Sequential DataOrdered observations where temporal/sequence order matters
RNNProcesses current input together with previous hidden state
Hidden StateLearned representation carrying information through the sequence
Recurrent ConnectionCarries information from one time step to the next
UnrollingRepresents recurrent computation explicitly across time
BPTTCalculates gradients by backpropagating through unrolled time steps
Bidirectional RNNUses both past and future sequence context
Stacked RNNUses multiple recurrent layers to increase network depth

19. Quick Recap

text
Sequential Data
→ Data where order matters.

RNN
→ Processes one time step at a time while maintaining hidden state.

Hidden State
→ Carries learned information from previous time steps.

Recurrent Connection
→ Transfers previous hidden information to the current step.

Unrolling
→ Expands the RNN across all time steps.

BPTT
→ Backpropagates gradients through those time steps.

Bidirectional RNN
→ Processes the sequence forward and backward.

Stacked RNN
→ Places multiple RNN layers on top of one another.

Final Mental Model

Architecture & Data Flow
                  SEQUENCE MODELS
                        |
                        v
                 Sequential Data
                        |
                        v
                       RNN
                        |
          +-------------+-------------+
          |                           |
          v                           v
    Hidden State               Recurrent Connection
          |                           |
          +-------------+-------------+
                        |
                        v
                    Unrolling
                        |
                        v
                       BPTT
                        |
             +----------+----------+
             |                     |
             v                     v
       Bidirectional          Stacked RNN
           RNN
       Past + Future         Multiple RNN Layers
         Context

The central idea to remember is:

text
Current Input
     +
Previous Hidden State
     ↓
     RNN
     ↓
Current Hidden State
     ↓
Next Time Step

That recurrent flow is what allows an RNN to model dependencies across a sequence.

Knowledge Checkpoint

14. Sequence Models & RNNs Checkpoint

Q1.What is the recurrent state update formula in a standard vanilla RNN at time step t?
Ah_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b_h)
Bh_t = x_t · W_xh + b
Ch_t = Softmax(h_{t-1} + x_t)
Dh_t = h_0 / t
Q2.Why do standard RNNs suffer from vanishing and exploding gradients over long sequences during BPTT?
ARepeated matrix multiplication by the same recurrent weight matrix W_hh across T time steps scales gradients by (W_hh)^T, causing exponential growth or decay to zero.
BBecause time steps cannot be executed in parallel on CPUs.
CBecause RNNs cannot process strings.
DBecause loss is only calculated at step 0.
Q3.What simple technique is universally applied to protect recurrent network training from exploding gradients?
AGradient Clipping (rescaling gradient vector if ||g|| exceeds a threshold c).
BSetting learning rate to zero on negative steps.
CUsing linear activations instead of tanh.
DDropping every second token in the sequence.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
13. Computer Vision Tasks (Detection, Segmentation, Tracking)
Next Lesson
15. Long Short-Term Memory Networks (LSTM)