Intermediate

24 min read

#LSTM#Cell State#Forget Gate#Input Gate#Output Gate#Sequences#PyTorch

15. Long Short-Term Memory Networks (LSTM)

Overcoming long-term dependencies: Cell state highway, forget gate, input gate, output candidate generation, and bidirectional LSTM architectures.

LSTM: Complete Notes (Beginner to Advanced)

Introduction#

Long Short-Term Memory (LSTM) is a specialized type of Recurrent Neural Network (RNN) designed to learn dependencies across sequences, especially when useful information needs to be carried across many time steps.

A standard RNN maintains a hidden state:

›xₜ + hₜ₋₁ → RNN → hₜ

LSTM adds a separate cell state and uses gates to control what information should be:

text
Forgotten
Added
Exposed

The core LSTM components are:

text
Cell State
Hidden State
Forget Gate
Input Gate
Candidate State
Output Gate

A useful high-level view is:

Architecture & Data Flow
Previous Cell State ────────────────────────────────┐
                                                    |
Previous Hidden State ──┐                           |
                        ↓                           ↓
Current Input ───────> LSTM Cell ─────────────> New Cell State
                        |
                        └──────────────────────> New Hidden State

The central idea is that the LSTM controls the flow of information using gates.

1. Long Short-Term Memory (LSTM)

Long Short-Term Memory (LSTM) is a gated recurrent neural network architecture.

It was introduced to address an important difficulty with traditional RNNs: learning useful dependencies over long sequences can be difficult because gradients may vanish or explode during Backpropagation Through Time (BPTT).

An LSTM introduces a memory pathway called the cell state and several gates that regulate information flow.

Standard RNN#

A simplified RNN update is:

Mathematical Formulation
hₜ = f(Wₓh xₜ + Wₕh hₜ₋₁ + b)

The same hidden state is responsible for carrying the recurrent information.

LSTM#

An LSTM maintains two main states:

›Cell State  → cₜ
Hidden State → hₜ

and uses gates to control them.

The high-level flow is:

Architecture & Data Flow
              Previous Cell State
                      |
                      v
                +-----------+
                |           |
Previous hₜ₋₁ →|   LSTM    |→ New hₜ
Current xₜ ───→ |   Cell    |
                |           |→ New cₜ
                +-----------+

Why LSTM?#

The important motivation is:

text
RNN
→ difficult to preserve useful information over long sequences

LSTM
→ explicit memory path + gates
→ better control over information flow

LSTM does not guarantee perfect long-term memory, but its architecture makes learning long-range dependencies easier than in a basic RNN in many settings.

2. LSTM Cell

An LSTM cell receives:

text
Current input: xₜ
Previous hidden state: hₜ₋₁
Previous cell state: cₜ₋₁

and produces:

›Current hidden state: hₜ
Current cell state: cₜ

The LSTM can be viewed as a sequence of controlled operations:

Architecture & Data Flow
xₜ + hₜ₋₁
      |
      +--> Forget Gate
      |
      +--> Input Gate
      |       |
      |       +--> Candidate State
      |
      +--> Update Cell State
      |
      +--> Output Gate
              |
              v
        New Hidden State

The main equations are:

Mathematical Formulation
fₜ = σ(W_f [hₜ₋₁, xₜ] + b_f)

iₜ = σ(W_i [hₜ₋₁, xₜ] + b_i)

c̃ₜ = tanh(W_c [hₜ₋₁, xₜ] + b_c)

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

oₜ = σ(W_o [hₜ₋₁, xₜ] + b_o)

hₜ = oₜ ⊙ tanh(cₜ)

Where:

Mathematical Formulation
σ   = sigmoid activation
tanh = hyperbolic tangent
⊙   = element-wise multiplication
[ ] = concatenation

The gates usually use sigmoid because their values lie between:

›0 and 1

This makes them useful as soft information-control mechanisms.

3. Cell State

The Cell State is the long-term memory pathway of an LSTM.

It is represented as:

›cₜ

The cell state carries information through the sequence while the gates determine what should be removed and what should be added.

Cell State Flow#

text
cₜ₋₁ ───────────────────────────────→ cₜ
          |                    |
          |                    |
          ↓                    ↓
     Forget Gate          Input Update

The update equation is:

Mathematical Formulation
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

There are two main operations:

›fₜ ⊙ cₜ₋₁

controls how much previous memory is retained.

›iₜ ⊙ c̃ₜ

controls how much new candidate information is written.

Therefore:

Mathematical Formulation
New Cell State
=
Retained Old Memory
+
Selected New Information

Why the Cell State Is Important#

The cell state provides a relatively direct pathway through time.

Conceptually:

›c₁ → c₂ → c₃ → c₄ → c₅

The gates modify this memory rather than completely recomputing it from scratch at every step.

This design helps LSTMs preserve useful information for longer periods.

Forgetting Information#

If:

Mathematical Formulation
fₜ ≈ 0

then:

Mathematical Formulation
fₜ ⊙ cₜ₋₁ ≈ 0

so most previous cell-state information is discarded.

If:

Mathematical Formulation
fₜ ≈ 1

then most of the previous cell state is retained.

4. Hidden State

The Hidden State is represented as:

›hₜ

It is the current output representation of the LSTM and is passed to the next time step.

The hidden state is calculated using:

Mathematical Formulation
hₜ = oₜ ⊙ tanh(cₜ)

Therefore, the hidden state depends on:

text
Current Cell State
+
Output Gate

Cell State vs Hidden State#

These two states are related but have different roles.

State	Symbol	Main Role
Cell State	`cₜ`	Carries long-term memory
Hidden State	`hₜ`	Current exposed/output representation

A useful intuition is:

text
Cell State
→ internal memory

Hidden State
→ information exposed by the LSTM at the current time step

This is an intuition rather than a strict separation of "long-term" and "short-term" information.

Flow#

text
Previous hₜ₋₁
       +
Current xₜ
       ↓
    LSTM Gates
       ↓
New cₜ
       ↓
Output Gate
       ↓
New hₜ

The new hidden state is also passed forward:

›hₜ → next LSTM step

5. Forget Gate

The Forget Gate decides how much of the previous cell state should be retained.

It is represented as:

›fₜ

The equation is:

Mathematical Formulation
fₜ = σ(W_f [hₜ₋₁, xₜ] + b_f)

The sigmoid produces values between:

›0 and 1

Therefore each component of the forget gate can be interpreted as a soft retention value.

Interpretation#

Mathematical Formulation
fₜ ≈ 0
→ forget most of that information

fₜ ≈ 1
→ retain most of that information

Example#

Suppose:

Mathematical Formulation
Previous memory:

cₜ₋₁ = [0.8, 0.6, 0.4]

and the forget gate produces:

Mathematical Formulation
fₜ = [1.0, 0.0, 0.7]

Then:

Mathematical Formulation
fₜ ⊙ cₜ₋₁
=
[1.0, 0.0, 0.7]
⊙
[0.8, 0.6, 0.4]

Result:

›[0.8, 0.0, 0.28]

So the second component is effectively forgotten while the first is retained.

Mental Model#

Architecture & Data Flow
Previous Cell State
        |
        v
   [Forget Gate]
        |
        v
How much to keep?
        |
        v
Retained Memory

6. Input Gate

The Input Gate controls how much new information should be written into the cell state.

It is represented as:

›iₜ

The equation is:

Mathematical Formulation
iₜ = σ(W_i [hₜ₋₁, xₜ] + b_i)

The sigmoid output lies between:

›0 and 1

Interpretation#

Mathematical Formulation
iₜ ≈ 0
→ write very little new information

iₜ ≈ 1
→ allow more new information to be written

However, the input gate alone does not determine the new content.

The actual proposed information comes from the candidate state.

This distinction is important:

text
Input Gate
→ how much to write

Candidate State
→ what could be written

7. Candidate State

The Candidate State is the new information that the LSTM proposes to add to the cell state.

It is commonly represented as:

›c̃ₜ

and calculated using:

Mathematical Formulation
c̃ₜ = tanh(W_c [hₜ₋₁, xₜ] + b_c)

The tanh activation produces values approximately in:

›[-1, 1]

Why Candidate State?#

The LSTM needs to determine:

›What new information should be considered?

The candidate state provides that proposed information.

Then the input gate determines how much of it should actually be written.

Architecture & Data Flow
Current Input + Previous Hidden State
                 |
                 v
          Candidate State
                 |
                 v
         Proposed New Memory
                 ^
                 |
           Input Gate
                 |
                 v
       How much to write?

Input Gate + Candidate State#

The actual contribution to the cell state is:

›iₜ ⊙ c̃ₜ

So:

Mathematical Formulation
Input Gate
×
Candidate State
=
New Information Added

8. Updating the Cell State

After calculating the forget gate, input gate, and candidate state, the LSTM updates its cell state.

The equation is:

Mathematical Formulation
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

This equation is the core of LSTM memory management.

Break it into two parts:

Part 1: Retain Previous Memory#

›fₜ ⊙ cₜ₋₁

The forget gate controls how much old memory remains.

Part 2: Add New Information#

›iₜ ⊙ c̃ₜ

The input gate controls how much candidate information is added.

Complete Interpretation#

Mathematical Formulation
New Memory
=
Old Memory × Forget Gate
+
Candidate × Input Gate

Or simply:

Mathematical Formulation
KEEP old information
+
WRITE useful new information
=
NEW cell state

9. Output Gate

The Output Gate controls how much of the current cell state should be exposed as the hidden state.

It is represented as:

›oₜ

The equation is:

Mathematical Formulation
oₜ = σ(W_o [hₜ₋₁, xₜ] + b_o)

The hidden state is then:

Mathematical Formulation
hₜ = oₜ ⊙ tanh(cₜ)

Interpretation#

›Output Gate
→ decides how much of the current memory is exposed

The cell state itself may contain information that should remain internal.

The output gate controls what becomes the current hidden representation.

Example#

Suppose:

Mathematical Formulation
tanh(cₜ) = [0.8, 0.5, -0.4]

and:

Mathematical Formulation
oₜ = [1.0, 0.2, 0.0]

Then:

Mathematical Formulation
hₜ = [1.0, 0.2, 0.0]
     ⊙
     [0.8, 0.5, -0.4]

Therefore:

Mathematical Formulation
hₜ = [0.8, 0.1, 0.0]

The output gate allowed some components to be exposed while suppressing others.

10. Complete LSTM Flow

All the components work together in a specific order.

Given:

text
xₜ
hₜ₋₁
cₜ₋₁

the LSTM performs:

Step 1: Forget Gate#

Mathematical Formulation
fₜ = σ(W_f [hₜ₋₁, xₜ] + b_f)

Decides:

›What old information should be retained?

Step 2: Input Gate#

Mathematical Formulation
iₜ = σ(W_i [hₜ₋₁, xₜ] + b_i)

Decides:

›How much new information should be written?

Step 3: Candidate State#

Mathematical Formulation
c̃ₜ = tanh(W_c [hₜ₋₁, xₜ] + b_c)

Creates:

›Candidate new information

Step 4: Cell State Update#

Mathematical Formulation
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

Combines:

text
Retained old memory
+
Selected new information

Step 5: Output Gate#

Mathematical Formulation
oₜ = σ(W_o [hₜ₋₁, xₜ] + b_o)

Decides:

›How much memory should be exposed?

Step 6: Hidden State#

Mathematical Formulation
hₜ = oₜ ⊙ tanh(cₜ)

Produces the current hidden representation.

Complete Diagram#

Architecture & Data Flow
                    xₜ
                    |
                    |
             +------+------+
             |             |
          hₜ₋₁            |
             |             |
             +------+------+
                    |
                    v
        +-----------------------+
        |       LSTM Cell       |
        |                       |
        |  +----------------+   |
        |  |  Forget Gate   |   |
        |  +----------------+   |
        |           |           |
        |           v           |
cₜ₋₁ ──────────────×───────────+
        |                       |
        |  +----------------+   |
        |  |  Input Gate    |   |
        |  +----------------+   |
        |           |           |
        |           v           |
        |  +----------------+   |
        |  | Candidate      |   |
        |  | State          |   |
        |  +----------------+   |
        |           |           |
        |           v           |
        |     New Cell State    |
        |           |           |
        |           v           |
        |  +----------------+   |
        |  |  Output Gate   |   |
        |  +----------------+   |
        |           |           |
        +-----------|-----------+
                    |
                    v
                   hₜ

11. LSTM Equations Together

For reference, all standard LSTM equations can be written together:

Mathematical Formulation
fₜ = σ(W_f [hₜ₋₁, xₜ] + b_f)

iₜ = σ(W_i [hₜ₋₁, xₜ] + b_i)

c̃ₜ = tanh(W_c [hₜ₋₁, xₜ] + b_c)

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

oₜ = σ(W_o [hₜ₋₁, xₜ] + b_o)

hₜ = oₜ ⊙ tanh(cₜ)

The flow is:

Architecture & Data Flow
xₜ + hₜ₋₁
      |
      +--> Forget Gate ──────────┐
      |                          |
      +--> Input Gate            |
      |        |                 |
      |        +--> Candidate    |
      |                          |
      +--------------------------+
                 |
                 v
             Cell State cₜ
                 |
                 v
             Output Gate
                 |
                 v
            Hidden State hₜ

12. Numerical Example

Consider a simplified scalar LSTM.

Suppose:

Mathematical Formulation
cₜ₋₁ = 0.8

and the gates and candidate state have already been calculated:

Mathematical Formulation
fₜ = 0.7
iₜ = 0.4
c̃ₜ = 0.5

The new cell state is:

Mathematical Formulation
cₜ = fₜ × cₜ₋₁ + iₜ × c̃ₜ

Substitute:

Mathematical Formulation
cₜ = 0.7 × 0.8 + 0.4 × 0.5

Mathematical Formulation
cₜ = 0.56 + 0.20

Mathematical Formulation
cₜ = 0.76

Now suppose:

Mathematical Formulation
oₜ = 0.6

The hidden state is:

Mathematical Formulation
hₜ = oₜ × tanh(cₜ)

Therefore:

Mathematical Formulation
hₜ = 0.6 × tanh(0.76)

Approximately:

Mathematical Formulation
tanh(0.76) ≈ 0.641

So:

Mathematical Formulation
hₜ ≈ 0.6 × 0.641

Mathematical Formulation
hₜ ≈ 0.385

The important sequence is:

text
Previous Memory
      ↓
Forget Gate
      ↓
Retained Memory
      +
Candidate × Input Gate
      ↓
New Cell State
      ↓
Output Gate
      ↓
New Hidden State

13. LSTM Parameter Structure

An LSTM has separate parameter sets for its four main transformations:

text
Forget Gate
Input Gate
Candidate State
Output Gate

For an input size:

›D

and hidden size:

›H

each gate transformation receives the concatenation:

›[hₜ₋₁, xₜ]

whose size is:

›H + D

Each transformation therefore has weights of approximately:

›H × (H + D)

and a bias of:

›H

There are four such transformations.

Therefore, the total parameter count for a standard LSTM layer is:

›4 × [H(H + D) + H]

or:

›4 × H × (H + D + 1)

Example#

Suppose:

Mathematical Formulation
Input size  = 10
Hidden size = 20

Then:

Mathematical Formulation
Parameters
=
4 × [20(20 + 10) + 20]

Mathematical Formulation
= 4 × [600 + 20]

Mathematical Formulation
= 4 × 620

Mathematical Formulation
= 2480

This is for one direction and one LSTM layer under the standard formulation.

14. Simple LSTM Implementation with PyTorch

PyTorch provides an nn.LSTM module.


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

lstm = nn.LSTM(
    input_size=10,
    hidden_size=20,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, (hidden, cell) = lstm(x)

print("Output shape:", output.shape)
print("Hidden shape:", hidden.shape)
print("Cell shape:", cell.shape)

Here:

Mathematical Formulation
batch_size = 4
sequence_length = 5
input_size = 10
hidden_size = 20

Therefore:

text
Input:
(4, 5, 10)

Output:
(4, 5, 20)

For one layer and one direction:

text
Hidden:
(1, 4, 20)

Cell:
(1, 4, 20)

The two returned states are:

›hidden → final hidden state
cell   → final cell state

15. LSTM with Multiple Layers

LSTMs can also be stacked.


🐍 PythonInteractive WebAssembly
lstm = nn.LSTM(
    input_size=10,
    hidden_size=20,
    num_layers=3,
    batch_first=True
)

x = torch.randn(4, 5, 10)

output, (hidden, cell) = lstm(x)

print(output.shape)
print(hidden.shape)
print(cell.shape)

For:

›3 layers
1 direction

the shapes are:

text
Output:
(4, 5, 20)

Hidden:
(3, 4, 20)

Cell:
(3, 4, 20)

The first dimension of hidden and cell represents:

›number of layers × number of directions

16. Bidirectional LSTM

Although this topic focuses on the LSTM mechanism itself, an LSTM can also be made bidirectional.

A bidirectional LSTM processes the sequence:

text
Forward:
x₁ → x₂ → x₃ → x₄

Backward:
x₄ → x₃ → x₂ → x₁

The representations from both directions are combined.

In PyTorch:


🐍 PythonInteractive WebAssembly
lstm = nn.LSTM(
    input_size=10,
    hidden_size=20,
    batch_first=True,
    bidirectional=True
)

For one layer:

Mathematical Formulation
Output feature size
=
2 × hidden_size
=
40

So for:

Mathematical Formulation
Input = (4, 5, 10)

the output is:

›(4, 5, 40)

The hidden and cell states have:

›2 × number_of_layers

as their first dimension.

17. RNN vs LSTM

A basic RNN has one primary recurrent hidden state:

›hₜ

An LSTM has:

text
hₜ
+
cₜ

and uses gates.

Feature	RNN	LSTM
Hidden State	Yes	Yes
Cell State	No	Yes
Forget Gate	No	Yes
Input Gate	No	Yes
Output Gate	No	Yes
Candidate State	Not as a separate gated component	Yes
Long-range dependency handling	More difficult in many cases	Often better
Architecture	Simpler	More complex
Parameters	Fewer	More

The important conceptual difference is:

text
RNN:
Previous hidden state + current input
                    ↓
                New hidden state

LSTM:
Previous hidden state + current input
                    ↓
             Multiple gates
                    ↓
          New cell + hidden state

18. Gate Mental Model

A simple way to remember the gates is:

text
FORGET GATE
→ What old information should I keep?

INPUT GATE
→ How much new information should I write?

CANDIDATE STATE
→ What new information could I write?

CELL STATE
→ What memory should I carry forward?

OUTPUT GATE
→ How much of that memory should I expose?

HIDDEN STATE
→ What representation do I output at this time step?

A compact memory formula is:

Mathematical Formulation
New Memory
=
Old Memory × Forget
+
Candidate × Input

Then:

Mathematical Formulation
Hidden State
=
tanh(New Memory) × Output

19. Complete LSTM Time-Step Flow

At every time step:

Architecture & Data Flow
Input xₜ
   +
Previous Hidden State hₜ₋₁
   +
Previous Cell State cₜ₋₁
          |
          v
     +----------+
     |  LSTM    |
     +----------+
          |
          +--> Forget Gate
          |
          +--> Input Gate
          |
          +--> Candidate State
          |
          v
     New Cell State cₜ
          |
          +--> Output Gate
          |
          v
     New Hidden State hₜ

Then the states move to the next time step:

›hₜ → hₜ₊₁
cₜ → cₜ₊₁

Therefore:

text
x₁ → LSTM → h₁, c₁
x₂ → LSTM → h₂, c₂
x₃ → LSTM → h₃, c₃
...

The same LSTM parameters are reused across time steps.

20. Summary

Component	Symbol	Main Function
Long Short-Term Memory	LSTM	Gated recurrent architecture
Cell State	`cₜ`	Carries memory through the sequence
Hidden State	`hₜ`	Current exposed representation/output
Forget Gate	`fₜ`	Controls how much old memory is retained
Input Gate	`iₜ`	Controls how much candidate information is written
Candidate State	`c̃ₜ`	Proposes new information for the cell state
Output Gate	`oₜ`	Controls how much cell-state information becomes hidden state

21. Quick Recap

text
LSTM
→ A gated RNN designed to handle sequence dependencies.

Cell State
→ Main memory pathway.

Hidden State
→ Current output representation.

Forget Gate
→ Controls what old memory to keep.

Input Gate
→ Controls how much new information to write.

Candidate State
→ Proposed new information.

Output Gate
→ Controls what information is exposed through the hidden state.

The complete mathematical flow is:

Mathematical Formulation
fₜ = σ(W_f [hₜ₋₁, xₜ] + b_f)

iₜ = σ(W_i [hₜ₋₁, xₜ] + b_i)

c̃ₜ = tanh(W_c [hₜ₋₁, xₜ] + b_c)

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ

oₜ = σ(W_o [hₜ₋₁, xₜ] + b_o)

hₜ = oₜ ⊙ tanh(cₜ)

Final Mental Model

Architecture & Data Flow
                 CURRENT INPUT xₜ
                        |
                        v
              +-------------------+
              |                   |
PREVIOUS hₜ₋₁ →       LSTM        |
              |                   |
              +-------------------+
                   |    |    |
                   |    |    |
                   v    v    v
                Forget Input Output
                 Gate   Gate  Gate
                   |     |     |
                   |     v     |
                   | Candidate |
                   |     |     |
                   +-----+     |
                         |     |
                         v     |
                   Cell State cₜ
                         |
                         v
                  tanh(cₜ)
                         |
                         ×
                    Output Gate
                         |
                         v
                  Hidden State hₜ
                         |
                         v
                   Next Time Step

The easiest way to remember an LSTM is:

text
FORGET
   ↓
decide what old memory to remove

INPUT + CANDIDATE
   ↓
decide what new information to add

CELL STATE
   ↓
carry the updated memory

OUTPUT
   ↓
decide what part of memory to expose

HIDDEN STATE
   ↓
pass the current representation forward

Knowledge Checkpoint

15. LSTM Checkpoint

Q1.What architectural innovation enables LSTMs to carry information across long time steps without gradient decay?

AThe Cell State (C_t) linear conveyor belt highway, modified only by additive gated interactions.

BReplacing matrix multiplication with attention weights.

CRunning backward passes before forward passes.

DDisabling all bias parameters.

Q2.What is the role of the Forget Gate (f_t = sigmoid(W_f · [h_{t-1}, x_t] + b_f)) in an LSTM cell?

AIt outputs values between 0 and 1 for each cell state component, deciding what proportion of past memory to retain (1) or erase (0).

BIt clears the GPU cache between epochs.

CIt deletes unseen vocabulary words from embedding tables.

DIt sets learning rate to zero.

Q3.Why are Bidirectional LSTMs (BiLSTM) advantageous for NLP tasks like Named Entity Recognition?

AThey run two independent LSTM layers (forward 1->T and backward T->1), providing full past and future context at each token step.

BThey train twice as fast on single-core CPUs.

CThey remove the need for token embeddings.

DThey allow autoregressive generation without prompt input.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

14. Sequence Models & Recurrent Neural Networks (RNN)

Next Lesson

16. Gated Recurrent Units (GRU)