∇GradientSchool
Intermediate
20 min read
#GRU#Reset Gate#Update Gate#Sequence Modeling#RNN#Efficiency

16. Gated Recurrent Units (GRU)

Streamlined gated sequence processing: Reset and update gate mechanisms, hidden state interpolation, computational efficiency, and GRU vs LSTM comparison.

GRU: Complete Notes (Beginner to Advanced)

Introduction#

Gated Recurrent Unit (GRU) is a type of recurrent neural network designed to process sequential data while controlling how information is retained and updated over time.

GRU was introduced as a simpler gated alternative to the Long Short-Term Memory (LSTM) architecture.

A basic RNN uses:

text
Current Input + Previous Hidden State
                ↓
               RNN
                ↓
        Current Hidden State

A GRU adds gates that control the hidden state:

Architecture & Data Flow
Current Input + Previous Hidden State
                ↓
          +-------------+
          |     GRU     |
          +-------------+
             |       |
             v       v
        Update Gate  Reset Gate
             |
             v
      Candidate Hidden State
             |
             v
      New Hidden State

The main GRU components are:

text
Update Gate
Reset Gate
Candidate Hidden State

Unlike LSTM, a standard GRU does not maintain a separate cell state.

1. Gated Recurrent Unit (GRU)

A Gated Recurrent Unit (GRU) is a gated recurrent architecture that maintains a hidden state and uses learned gates to regulate information flow.

At time step t, a GRU receives:

›xₜ
hₜ₋₁

and produces:

›hₜ

The standard GRU equations are:

Mathematical Formulation
zₜ = σ(W_z xₜ + U_z hₜ₋₁ + b_z)

rₜ = σ(W_r xₜ + U_r hₜ₋₁ + b_r)

h̃ₜ = tanh(W_h xₜ + U_h (rₜ ⊙ hₜ₋₁) + b_h)

hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

Where:

Mathematical Formulation
xₜ       = current input
hₜ₋₁     = previous hidden state
h̃ₜ       = candidate hidden state
hₜ       = new hidden state

zₜ       = update gate
rₜ       = reset gate

σ        = sigmoid
tanh     = hyperbolic tangent
⊙        = element-wise multiplication

High-Level Flow#

Architecture & Data Flow
                xₜ
                 |
                 v
          +--------------+
          |              |
      hₜ₋₁|     GRU      |
          |              |
          +--------------+
             |        |
             v        v
        Update Gate  Reset Gate
             |        |
             |        v
             |   Candidate State
             |        |
             +--------+
                  |
                  v
             Hidden State

Main Idea#

The GRU asks two important questions:

text
Update Gate:
How much should the new candidate replace the old hidden state?

Reset Gate:
How much of the previous hidden state should be considered when creating the candidate?

This allows the GRU to control memory without using a separate cell state.

2. Update Gate

The Update Gate determines how much of the previous hidden state should be retained versus how much should be replaced by the candidate hidden state.

It is commonly represented as:

›zₜ

The equation is:

Mathematical Formulation
zₜ = σ(W_z xₜ + U_z hₜ₋₁ + b_z)

Because sigmoid produces values between:

›0 and 1

each component of the update gate acts like a soft control value.

Interpretation#

Using the common GRU formulation:

Mathematical Formulation
zₜ ≈ 0
→ retain more of the previous hidden state

zₜ ≈ 1
→ use more of the candidate hidden state

The final hidden-state update is:

Mathematical Formulation
hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

Therefore:

Mathematical Formulation
New Hidden State
=
Old Hidden State × (1 - Update Gate)
+
Candidate × Update Gate

Example#

Suppose:

Mathematical Formulation
hₜ₋₁ = 0.8
h̃ₜ  = 0.2
zₜ   = 0.25

Then:

Mathematical Formulation
hₜ = (1 - 0.25)(0.8) + 0.25(0.2)
Mathematical Formulation
hₜ = 0.75(0.8) + 0.25(0.2)
Mathematical Formulation
hₜ = 0.60 + 0.05
Mathematical Formulation
hₜ = 0.65

The new hidden state is closer to the old hidden state because the update gate is relatively small.

Mental Model#

Architecture & Data Flow
Previous Hidden State
        |
        +--------------------+
        |                    |
        |              (1 - zₜ)
        |                    |
        v                    v
                         Retained
                         Information

Candidate Hidden State
        |
        | zₜ
        v
    New Information

        \                 /
         \               /
          +-------------+
                 |
                 v
             hₜ

3. Reset Gate

The Reset Gate determines how much of the previous hidden state should be used when creating the candidate hidden state.

It is represented as:

›rₜ

The equation is:

Mathematical Formulation
rₜ = σ(W_r xₜ + U_r hₜ₋₁ + b_r)

Again:

›0 ≤ rₜ ≤ 1

Interpretation#

Mathematical Formulation
rₜ ≈ 0
→ largely ignore the previous hidden state when forming the candidate

rₜ ≈ 1
→ use more of the previous hidden state

The reset gate appears in the candidate calculation:

Mathematical Formulation
h̃ₜ = tanh(W_h xₜ + U_h (rₜ ⊙ hₜ₋₁) + b_h)

The key operation is:

›rₜ ⊙ hₜ₋₁

This controls how much previous information reaches the candidate computation.

Example#

Suppose:

Mathematical Formulation
hₜ₋₁ = 0.8
rₜ = 0.2

Then:

Mathematical Formulation
rₜ × hₜ₋₁
=
0.2 × 0.8
=
0.16

Only a small portion of the previous hidden information is passed into the candidate calculation.

Mental Model#

Architecture & Data Flow
Previous Hidden State
        |
        v
   [Reset Gate]
        |
        v
How much previous
information to use?
        |
        v
Candidate Calculation

4. Candidate Hidden State

The Candidate Hidden State is the proposed new hidden representation.

It is commonly represented as:

›h̃ₜ

and calculated as:

Mathematical Formulation
h̃ₜ = tanh(W_h xₜ + U_h (rₜ ⊙ hₜ₋₁) + b_h)

The candidate uses:

text
Current Input
+
Reset-controlled Previous Hidden State

Why the Reset Gate Matters Here#

The reset gate controls the previous hidden state before it enters the candidate computation.

Architecture & Data Flow
hₜ₋₁
   |
   v
rₜ ⊙ hₜ₋₁
   |
   v
Candidate computation
   |
   v
h̃ₜ

Therefore:

text
Reset Gate
→ controls previous information used to create candidate

Candidate Hidden State
→ represents the proposed new information

Candidate Range#

Because tanh is used:

›-1 ≤ h̃ₜ ≤ 1

for each component.

5. Updating the Hidden State

Once the update gate and candidate hidden state are available, the GRU computes the new hidden state.

The standard formulation used here is:

Mathematical Formulation
hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

This can be understood as a weighted interpolation between:

›Previous Hidden State

and:

›Candidate Hidden State

If Update Gate Is Near Zero#

Mathematical Formulation
zₜ ≈ 0

Then:

Mathematical Formulation
hₜ ≈ hₜ₋₁

So the old hidden state is mostly preserved.

If Update Gate Is Near One#

Mathematical Formulation
zₜ ≈ 1

Then:

Mathematical Formulation
hₜ ≈ h̃ₜ

So the candidate largely replaces the old hidden state.

Important Note#

Different texts and implementations may use an alternative update-gate convention:

Mathematical Formulation
hₜ = zₜ ⊙ hₜ₋₁ + (1 - zₜ) ⊙ h̃ₜ

In that convention, the interpretation of zₜ is reversed.

Therefore, when studying a specific implementation, always check its exact equations.

The equations in these notes use:

Mathematical Formulation
hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

6. Complete GRU Flow

At every time step, the GRU performs the following operations.

Step 1: Calculate Update Gate#

Mathematical Formulation
zₜ = σ(W_z xₜ + U_z hₜ₋₁ + b_z)

Question:

›How much should the candidate replace the old hidden state?

Step 2: Calculate Reset Gate#

Mathematical Formulation
rₜ = σ(W_r xₜ + U_r hₜ₋₁ + b_r)

Question:

›How much previous information should influence the candidate?

Step 3: Calculate Candidate Hidden State#

Mathematical Formulation
h̃ₜ = tanh(W_h xₜ + U_h (rₜ ⊙ hₜ₋₁) + b_h)

Question:

›What new hidden representation could be created?

Step 4: Calculate New Hidden State#

Mathematical Formulation
hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

Question:

›How much old state and candidate state should be combined?

Complete Diagram#

Architecture & Data Flow
                     xₜ
                      |
                      |
              +-------+-------+
              |               |
              v               v
        Update Gate       Reset Gate
             zₜ               rₜ
              |               |
              |               v
              |        rₜ ⊙ hₜ₋₁
              |               |
              |               v
              |       Candidate State
              |              h̃ₜ
              |               |
              |               |
              +-------+-------+
                      |
                      v
            Combine old + candidate
                      |
                      v
                     hₜ

7. Numerical Example

Consider a simplified scalar GRU.

Suppose:

Mathematical Formulation
hₜ₋₁ = 0.8

and the gates and candidate have already been calculated:

Mathematical Formulation
zₜ = 0.3
rₜ = 0.2
h̃ₜ = 0.5

The reset gate has already been used to create the candidate.

Now calculate the new hidden state:

Mathematical Formulation
hₜ = (1 - zₜ)hₜ₋₁ + zₜh̃ₜ

Substitute:

Mathematical Formulation
hₜ = (1 - 0.3)(0.8) + 0.3(0.5)
Mathematical Formulation
hₜ = 0.7(0.8) + 0.3(0.5)
Mathematical Formulation
hₜ = 0.56 + 0.15
Mathematical Formulation
hₜ = 0.71

Therefore:

Mathematical Formulation
New Hidden State = 0.71

The previous state contributes strongly because:

Mathematical Formulation
zₜ = 0.3

is relatively small.

8. GRU vs Basic RNN

A basic RNN performs a direct recurrent update:

Mathematical Formulation
hₜ = tanh(Wₓh xₜ + Wₕh hₜ₋₁ + b)

A GRU introduces gates:

text
Update Gate
Reset Gate
Candidate Hidden State

Comparison:

FeatureBasic RNNGRU
Hidden StateYesYes
Separate Cell StateNoNo
Update GateNoYes
Reset GateNoYes
Gated Memory ControlNoYes
ArchitectureSimplerMore complex
Long-range dependency handlingMore difficult in many casesOften better

The GRU provides explicit control over how previous and new information are combined.

9. LSTM vs GRU

LSTM and GRU are both gated recurrent architectures.

The major architectural difference is:

text
LSTM
→ Hidden State + Cell State
→ Three main gates

GRU
→ Hidden State only
→ Two main gates

LSTM#

An LSTM maintains:

›cₜ → Cell State
hₜ → Hidden State

and commonly uses:

text
Forget Gate
Input Gate
Output Gate
Candidate State

GRU#

A GRU maintains:

›hₜ → Hidden State

and uses:

text
Update Gate
Reset Gate
Candidate Hidden State

Direct Comparison#

FeatureLSTMGRU
Hidden StateYesYes
Cell StateYesNo
Forget GateYesNo
Input GateYesNo
Output GateYesNo
Update GateNoYes
Reset GateNoYes
Candidate StateYesYes
Number of main gates32
Separate memory stateYesNo
Typical parameter countHigherLower
ArchitectureMore complexSimpler

Parameter Count#

For input size:

›D

and hidden size:

›H

a standard LSTM has four parameterized transformations:

›4 × [H(H + D) + H]

A standard GRU has three:

›3 × [H(H + D) + H]

Therefore, approximately:

Mathematical Formulation
GRU parameters
≈ 3/4 of LSTM parameters

when comparing otherwise equivalent single-layer, single-direction configurations.

The exact parameter count in a framework can differ depending on implementation details and optional biases.

10. LSTM and GRU Memory Mechanisms

The two architectures manage memory differently.

LSTM#

LSTM has an explicit cell state:

Architecture & Data Flow
cₜ₋₁
  |
  v
Forget + Add Information
  |
  v
cₜ
  |
  v
Output Gate
  |
  v
hₜ

GRU#

GRU directly updates the hidden state:

Architecture & Data Flow
hₜ₋₁
   |
   +----> Reset-controlled candidate
   |
   +----> Update-controlled retention
   |
   v
  hₜ

Therefore:

text
LSTM:
separate memory pathway + hidden state

GRU:
single hidden-state pathway

11. LSTM vs GRU: Conceptual Mapping

Some concepts can be loosely related, but they are not exact one-to-one equivalents.

text
LSTM:
Forget Gate
Input Gate
Candidate State
Output Gate
Cell State
Hidden State

versus:

text
GRU:
Update Gate
Reset Gate
Candidate Hidden State
Hidden State

A useful high-level comparison is:

text
LSTM
├── Decide what old memory to retain
├── Decide what new information to write
├── Maintain separate cell memory
└── Decide what to expose

GRU
├── Decide how much old/new state to combine
├── Decide how much previous state affects candidate
└── Maintain one hidden state

Do not treat:

Mathematical Formulation
LSTM Forget Gate = GRU Reset Gate

as an exact equivalence.

Their roles are related conceptually to information control, but their equations and positions in the computation are different.

12. When GRU Can Be Useful

GRUs can be attractive when:

text
A recurrent architecture is appropriate
+
A simpler architecture is preferred
+
Lower parameter count is useful

Potential advantages include:

  • Fewer parameters than a comparable standard LSTM
  • Simpler internal state structure
  • Potentially faster training or inference depending on implementation and workload
  • Useful performance on many sequence tasks

There is no universal rule that GRU is always better or faster than LSTM.

The appropriate choice depends on:

text
Dataset
Sequence length
Task
Model size
Hardware
Optimization
Training behavior

13. Simple GRU Implementation with PyTorch

PyTorch provides an nn.GRU module.

🐍 Python
import torch
import torch.nn as nn

gru = nn.GRU(
    input_size=10,
    hidden_size=20,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, hidden = gru(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

For:

Mathematical Formulation
batch_size = 4
sequence_length = 5
input_size = 10
hidden_size = 20

the shapes are:

text
Input:
(4, 5, 10)

Output:
(4, 5, 20)

Hidden:
(1, 4, 20)

The final hidden state contains the last hidden representation for the sequence.

14. Stacked GRU

GRU layers can be stacked using num_layers.

🐍 Python
gru = nn.GRU(
    input_size=10,
    hidden_size=20,
    num_layers=3,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, hidden = gru(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

For:

›3 layers
1 direction

the shapes are:

text
Output:
(4, 5, 20)

Hidden:
(3, 4, 20)

The first hidden-state dimension represents:

›number of layers × number of directions

15. Bidirectional GRU

A GRU can also process sequences in both directions.

🐍 Python
gru = nn.GRU(
    input_size=10,
    hidden_size=20,
    batch_first=True,
    bidirectional=True
)

x = torch.randn(4, 5, 10)

output, hidden = gru(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)

For one bidirectional layer:

Mathematical Formulation
Number of directions = 2

Therefore:

text
Output:
(4, 5, 40)

Hidden:
(2, 4, 20)

The output feature dimension is:

›2 × hidden_size

16. Complete GRU Mental Model

The entire GRU can be remembered using three questions:

text
1. UPDATE GATE
   How much should I replace the old hidden state?

2. RESET GATE
   How much of the old hidden state should influence
   the candidate?

3. CANDIDATE HIDDEN STATE
   What new representation should I propose?

Then:

Architecture & Data Flow
Previous Hidden State
        |
        +-----------------------------+
        |                             |
        |                         Reset Gate
        |                             |
        |                             v
        |                     Candidate State
        |                             |
        |                             |
        +----> Update Gate -----------+
                       |
                       v
                New Hidden State

17. Summary

ConceptMeaning
GRUGated recurrent architecture for sequential data
Update GateControls the balance between previous and candidate hidden state
Reset GateControls how much previous hidden information contributes to the candidate
Candidate Hidden StateProposed new hidden representation
Hidden StateGRU's main recurrent memory/output representation
LSTMGated recurrent architecture with separate cell and hidden states
GRU vs LSTMGRU uses fewer gates and no separate cell state

18. Quick Recap

text
GRU
→ A simpler gated alternative to LSTM.

Update Gate
→ Controls how much candidate information replaces old hidden information.

Reset Gate
→ Controls how much previous hidden information is used to form the candidate.

Candidate Hidden State
→ Proposed new hidden representation.

Hidden State
→ The GRU's recurrent state carried to the next time step.

LSTM vs GRU
→ LSTM has a separate cell state and three main gates.
→ GRU has one hidden state and two main gates.
→ GRU generally has fewer parameters.

The core GRU equations are:

Mathematical Formulation
zₜ = σ(W_z xₜ + U_z hₜ₋₁ + b_z)

rₜ = σ(W_r xₜ + U_r hₜ₋₁ + b_r)

h̃ₜ = tanh(W_h xₜ + U_h (rₜ ⊙ hₜ₋₁) + b_h)

hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ

Final Mental Model

Architecture & Data Flow
                     GRU
                      |
          +-----------+-----------+
          |                       |
          v                       v
     UPDATE GATE             RESET GATE
          |                       |
          |                       v
          |              Previous Hidden State
          |                       |
          |                       v
          |              Candidate Hidden State
          |                       |
          +-----------+-----------+
                      |
                      v
             New Hidden State hₜ
                      |
                      v
                Next Time Step

The simplest way to remember the difference is:

text
LSTM:
"Maintain a separate memory cell and control
forgetting, writing, and exposing."

GRU:
"Use gates to control how the previous hidden
state and a new candidate are combined."
Knowledge Checkpoint

16. GRU Checkpoint

Q1.How does a Gated Recurrent Unit (GRU) simplify the standard LSTM architecture?
AIt eliminates the separate Cell State (merging it into hidden state h_t) and replaces the 3 gates with just 2: Update Gate (z) and Reset Gate (r).
BIt removes all activation functions.
CIt replaces all matrices with scalar floats.
DIt disables the backward pass entirely.
Q2.What does the GRU hidden state update equation h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h~_t represent?
AA linear interpolation between previous hidden state h_{t-1} and candidate new state h~_t controlled by update gate z_t.
BA projection onto the unit sphere.
CA cross-product of inputs.
DA discrete step function.
Q3.In what practical scenarios is GRU often preferred over LSTM?
AWhen training smaller datasets or working under constrained memory/latency budgets, since GRUs have ~25% fewer parameters and train faster.
BWhen sequences exceed 100,000 steps.
CWhen working strictly with image classification.
DWhen using non-differentiable loss functions.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
15. Long Short-Term Memory Networks (LSTM)
Next Lesson
17. Attention Mechanisms (Bahdanau, Luong, Scaled Dot-Product)