∇GradientSchool
Advanced
100–150 min read
#Time Series#Deep Learning#RNN#LSTM#GRU#Sequence Modeling#TensorFlow#Keras#Forecasting

Deep Learning for Time Series: RNNs, LSTMs & GRUs

A detailed practical and conceptual guide to sequence-based deep learning for time series, covering RNNs, LSTMs, GRUs, sequence windows, 3D tensors, training, forecasting, evaluation, and common pitfalls.

Deep Learning for Time Series: RNNs, LSTMs & GRUs

1. Learning Objectives#

By the end of this notebook, you should be able to:

  • Explain why sequence data requires different modeling strategies.
  • Understand the limitations of ordinary feed-forward neural networks for time series.
  • Explain the basic architecture of Recurrent Neural Networks.
  • Understand hidden states and recurrent connections.
  • Understand vanishing and exploding gradients.
  • Explain why LSTMs were introduced.
  • Understand the LSTM cell and its gates.
  • Explain GRUs and how they differ from LSTMs.
  • Convert a time series into supervised sequences.
  • Build 3D tensors with shape (samples, timesteps, features).
  • Scale time-series data correctly without leakage.
  • Build RNN, LSTM, and GRU forecasting models with TensorFlow/Keras.
  • Train models using chronological validation.
  • Generate one-step and multi-step forecasts.
  • Evaluate and visualize deep-learning forecasts.
  • Recognize overfitting and apply appropriate regularization.
  • Understand when deep learning is useful and when simpler models may be preferable.

2. From Tree-Based ML to Sequence Learning

In Notebook 3, we transformed a time series into tabular features:

Architecture & Data Flow
Time Series
 |
 v
Lag Features
Rolling Features
Calendar Features
 |
 v
Tabular Dataset
 |
 v
Random Forest / XGBoost / LightGBM
 |
 v
Forecast

This approach is powerful, but the temporal structure is represented explicitly through engineered features.

Deep learning provides another approach:

Architecture & Data Flow
Time Series
 |
 v
Sequence Windows
 |
 v
Neural Network
 |
 v
Learned Temporal Representation
 |
 v
Forecast

Instead of manually specifying every useful relationship, recurrent networks can learn representations of sequences.

3. Why Standard Feed-Forward Networks Are Limited

A standard feed-forward neural network treats each input as a fixed vector.

For example:

›Input:
[lag_1, lag_2, lag_3, lag_4]

It can learn relationships among these values.

But sequence problems have an additional concept:

The order and repeated processing of observations matter.

Compare:

›[10, 20, 30]

with:

›[30, 20, 10]

A feed-forward model can process both as vectors, but it does not inherently maintain a sequential state that evolves from one timestep to the next.

Recurrent networks introduce this temporal state.

4. Sequence Modeling Intuition

Imagine reading:

›"The company reported strong..."

When you encounter the next word, information from previous words can help interpret it.

Similarly, when forecasting:

text
Yesterday
Last week
Previous month
Current observation

previous observations can provide context for the next prediction.

A recurrent network maintains a hidden representation of previous information.

5. The Basic RNN

A simple recurrent neural network can be expressed as:

ht=tanh⁡(Wxxt+Whht−1+bh)h_t = \tanh( W_xx_t + W_hh_{t-1} + b_h )

and:

yt=Wyht+byy_t = W_yh_t+b_y

where:

  • xtx_t = input at time tt
  • hth_t = hidden state at time tt
  • ht−1h_{t-1} = previous hidden state
  • WxW_x = input weights
  • WhW_h = recurrent weights
  • WyW_y = output weights
  • bb = biases

The important idea is:

ht=f(xt,ht−1)h_t = f(x_t,h_{t-1})

The current hidden state depends on:

  • Current input
  • Previous hidden state

6. RNN Unrolled Through Time

Conceptually:

Architecture & Data Flow
x1 x2 x3 x4
| | | |
v v v v
[RNN] -> [RNN] -> [RNN] -> [RNN]
 | | | |
 h1 h2 h3 h4

The same network parameters are reused at every timestep.

This parameter sharing allows the network to process sequences of different lengths.

7. Hidden State

The hidden state acts as a learned representation of previous information.

Conceptually:

Architecture & Data Flow
Current input
 +
Previous memory
 |
 v
New memory

At time tt:

ht=f(xt,ht−1)h_t=f(x_t,h_{t-1})

At the next timestep:

ht+1=f(xt+1,ht)h_{t+1}=f(x_{t+1},h_t)

Therefore, information can flow through time.

8. The Long-Term Dependency Problem

A basic RNN can struggle to retain information across long sequences.

For example:

Architecture & Data Flow
Event at t = 1
 |
 |
 |
 |
 |
Prediction at t = 100

The information may become difficult to preserve.

This is related to vanishing and exploding gradients during training.

9. Vanishing Gradients

During backpropagation through time, gradients are repeatedly multiplied through recurrent transformations.

If the effective gradient magnitude repeatedly becomes smaller than 1:

0.5×0.5×0.5×⋯0.5 \times 0.5 \times 0.5 \times \cdots

it can become extremely small.

This causes earlier timesteps to receive very little learning signal.

The network then struggles to learn long-term dependencies.

10. Exploding Gradients

The opposite can also occur.

If gradients repeatedly grow:

2×2×2×⋯2 \times 2 \times 2 \times \cdots

they can become extremely large.

This can lead to:

  • Unstable training
  • Extremely large parameter updates
  • Numerical problems

Gradient clipping can help control exploding gradients.

Example:

🐍 Python
from tensorflow.keras.optimizers import Adam

optimizer = Adam(
 learning_rate=0.001,
 clipnorm=1.0
)

11. Why LSTM?

LSTM stands for:

Long Short-Term Memory

LSTM networks were designed to improve the ability of recurrent networks to learn long-term dependencies.

Instead of relying only on a simple hidden-state update, an LSTM introduces a memory cell and gates.

The gates control:

  • What information to forget
  • What new information to store
  • What information to expose

12. LSTM Cell Intuition

A simplified LSTM cell:

Architecture & Data Flow
 ┌─────────────────┐
Previous memory ->| Forget Gate |
 └─────────────────┘
 |
 v
Input ----------> Update / Input Gate
 |
 v
 Cell State
 |
 v
 Output Gate
 |
 v
 Hidden State

The model learns which information should be retained or discarded.

13. LSTM Mathematical Formulation

Given:

  • Input xtx_t
  • Previous hidden state ht−1h_{t-1}
  • Previous cell state ct−1c_{t-1}

the forget gate is:

ft=σ(Wf[ht−1,xt]+bf)f_t = \sigma( W_f[h_{t-1},x_t]+b_f )

The input gate is:

it=σ(Wi[ht−1,xt]+bi)i_t = \sigma( W_i[h_{t-1},x_t]+b_i )

Candidate cell state:

c~t=tanh⁡(Wc[ht−1,xt]+bc)\tilde{c}_t = \tanh( W_c[h_{t-1},x_t]+b_c )

Cell state:

ct=ft⊙ct−1+it⊙c~tc_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t

Output gate:

ot=σ(Wo[ht−1,xt]+bo)o_t = \sigma( W_o[h_{t-1},x_t]+b_o )

Hidden state:

ht=ot⊙tanh⁡(ct)h_t = o_t \odot \tanh(c_t)

where:

  • σ\sigma = sigmoid activation
  • ⊙\odot = element-wise multiplication

14. LSTM Gates Intuition

The mathematics is useful, but the intuition is more important initially.

Forget Gate#

Question:

What old information should we discard?

Input Gate#

Question:

What new information should we store?

Cell State#

Question:

What information should flow through the sequence?

Output Gate#

Question:

What information should be exposed as the current hidden representation?

This gated memory mechanism helps LSTMs handle longer dependencies than basic RNNs in many situations.

15. GRU

GRU stands for:

Gated Recurrent Unit

GRUs are another gated recurrent architecture.

They are generally simpler than LSTMs.

A GRU uses two main gates:

  • Update gate
  • Reset gate

16. GRU Intuition

Update Gate#

Controls how much previous information should be retained versus replaced.

Reset Gate#

Controls how much previous state contributes to the candidate state.

Conceptually:

Architecture & Data Flow
Previous state
 |
 +------> Reset Gate
 |
 +------> Update Gate
 |
 v
New hidden state

GRUs often have fewer parameters than LSTMs.

17. LSTM vs GRU

CharacteristicLSTMGRU
Memory mechanismCell + hidden stateHidden state
Main gatesForget, Input, OutputUpdate, Reset
ComplexityHigherLower
ParametersMoreFewer
Training speedOften slowerOften faster
Long dependenciesStrongStrong
Practical performanceOften excellentOften excellent

There is no universal winner.

The correct approach is empirical evaluation.

18. Choosing Between RNN, LSTM, and GRU

A practical approach:

Architecture & Data Flow
Start with baseline
 |
 v
Try simple RNN
 |
 v
Try LSTM
 |
 v
Try GRU
 |
 v
Compare validation performance

If a simple RNN performs adequately, its simplicity may be valuable.

If long dependencies matter, LSTM or GRU may provide better performance.

19. Sequence Windows

Neural networks need sequences rather than isolated rows.

Suppose the time series is:

›10, 20, 30, 40, 50, 60

With:

Mathematical Formulation
lookback = 3

we can create:

Architecture & Data Flow
[10, 20, 30] -> 40
[20, 30, 40] -> 50
[30, 40, 50] -> 60

Each input contains three timesteps.

20. Formal Sequence Representation

For lookback LL:

Xt=[yt−L+1,…,yt]X_t = [ y_{t-L+1}, \ldots, y_t ]

Target:

yt+1y_{t+1}

This is a one-step-ahead forecasting setup.

21. Three-Dimensional Input

For multivariate time series, the neural-network input normally has:

›(samples, timesteps, features)

For example:

›(1000, 30, 5)

means:

  • 1000 training sequences
  • 30 timesteps per sequence
  • 5 features per timestep

This shape is fundamental to recurrent neural networks.

22. Create Sequence Windows

🐍 Python
def create_sequences(
 data,
 lookback,
 target_index=0
):
 X = []
 y = []

 for i in range(
 lookback,
 len(data)
 ):
 X.append(
 data[i-lookback:i]
 )

 y.append(
 data[i, target_index]
 )

 return np.array(X), np.array(y)

Example:

🐍 Python
values = np.arange(1, 11).reshape(-1, 1)

X, y = create_sequences(
 values,
 lookback=3
)

print(X.shape)
print(y.shape)

Expected conceptually:

›X -> (7, 3, 1)
y -> (7,)

23. Multivariate Sequences

Suppose each timestep has:

text
sales
price
temperature
promotion

Then:

Mathematical Formulation
features = 4

A sequence with a 30-day lookback becomes:

›(30, 4)

A batch becomes:

›(samples, 30, 4)

This allows the network to learn relationships across both:

  • Time
  • Features

24. Scaling Time Series Data

Neural networks often train more effectively when numerical inputs are on comparable scales.

A common choice is:

🐍 Python
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

or:

🐍 Python
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

The choice depends on the data and modeling setup.

25. Critical Rule: Scale Without Leakage

Do not fit the scaler on the entire dataset before splitting.

Incorrect:

🐍 Python
scaler.fit_transform(
 df[["sales"]]
)

before the train/test split.

Correct:

🐍 Python
train_values = train[["sales"]]

scaler.fit(train_values)

train_scaled = scaler.transform(
 train_values
)

test_scaled = scaler.transform(
 test[["sales"]]
)

The scaler must learn its parameters from training data only.

26. Chronological Split

Example:

🐍 Python
split_1 = int(
 len(df) * 0.70
)

split_2 = int(
 len(df) * 0.85
)

train = df.iloc[:split_1]

validation = df.iloc[
 split_1:split_2
]

test = df.iloc[
 split_2:
]

This preserves temporal order.

27. Build a Practice Dataset

🐍 Python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)

dates = pd.date_range(
 "2021-01-01",
 periods=1000,
 freq="D"
)

n = len(dates)

trend = np.linspace(
 100,
 200,
 n
)

weekly = (
 15 *
 np.sin(
 2 * np.pi * np.arange(n) / 7
 )
)

yearly = (
 10 *
 np.sin(
 2 * np.pi * np.arange(n) / 365
 )
)

noise = np.random.normal(
 0,
 5,
 n
)

sales = (
 trend
 + weekly
 + yearly
 + noise
)

df = pd.DataFrame({
 "date": dates,
 "sales": sales
})

df = (
 df
 .set_index("date")
)

df.head()

28. Visualize the Dataset

🐍 Python
df["sales"].plot(
 figsize=(14, 5)
)

plt.title(
 "Synthetic Daily Sales"
)

plt.xlabel("Date")
plt.ylabel("Sales")

plt.show()

Before training a neural network, always understand the data visually.

29. Train/Validation/Test Data

🐍 Python
n = len(df)

train_end = int(
 n * 0.70
)

validation_end = int(
 n * 0.85
)

train = df.iloc[
 :train_end
]

validation = df.iloc[
 train_end:validation_end
]

test = df.iloc[
 validation_end:
]

Visualize the split:

🐍 Python
plt.figure(figsize=(14, 5))

plt.plot(
 train.index,
 train["sales"],
 label="Train"
)

plt.plot(
 validation.index,
 validation["sales"],
 label="Validation"
)

plt.plot(
 test.index,
 test["sales"],
 label="Test"
)

plt.legend()

plt.show()

30. Scaling the Target

🐍 Python
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

train_scaled = scaler.fit_transform(
 train[["sales"]]
)

validation_scaled = scaler.transform(
 validation[["sales"]]
)

test_scaled = scaler.transform(
 test[["sales"]]
)

The validation and test sets are transformed using the training scaler.

31. Create Training Sequences

🐍 Python
lookback = 30

X_train, y_train = create_sequences(
 train_scaled,
 lookback=lookback
)

X_validation, y_validation = create_sequences(
 validation_scaled,
 lookback=lookback
)

X_test, y_test = create_sequences(
 test_scaled,
 lookback=lookback
)

Inspect:

🐍 Python
print(
 "X_train:",
 X_train.shape
)

print(
 "y_train:",
 y_train.shape
)

The expected structure is:

text
X_train:
(samples, 30, 1)

y_train:
(samples,)

32. Important Boundary Consideration

The simple sequence construction above treats each split independently.

That means the first validation sequence cannot use the final 30 training observations.

For some forecasting experiments, it is more realistic to allow the validation/test sequence context to include historical observations immediately before the split, because those values would genuinely be known at prediction time.

For example:

Architecture & Data Flow
Training history
 |
 +--> Last 30 observations
 |
 v
Validation first prediction

This requires careful construction.

The important principle is:

Historical context before a forecast origin can be used if it would actually be available at that forecast origin.

33. Building an RNN

TensorFlow/Keras provides recurrent layers directly.

🐍 Python
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import SimpleRNN, Dense

Model:

🐍 Python
rnn_model = Sequential([
 SimpleRNN(
 64,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dense(1)
])

Compile:

🐍 Python
rnn_model.compile(
 optimizer="adam",
 loss="mse"
)

34. Training the RNN

🐍 Python
history = rnn_model.fit(
 X_train,
 y_train,
 validation_data=(
 X_validation,
 y_validation
 ),
 epochs=30,
 batch_size=32,
 shuffle=False
)

For time-series sequence training, keeping shuffle=False can make the training process more consistent with chronological ordering, especially when stateful behavior or temporal interpretation is important.

For standard stateless recurrent layers, shuffling independent training windows does not inherently leak future information, but chronological ordering remains a useful convention.

35. Training Curves

🐍 Python
plt.figure(figsize=(10, 5))

plt.plot(
 history.history["loss"],
 label="Training Loss"
)

plt.plot(
 history.history["val_loss"],
 label="Validation Loss"
)

plt.title(
 "RNN Training History"
)

plt.xlabel("Epoch")
plt.ylabel("MSE")

plt.legend()

plt.show()

Interpretation:

Architecture & Data Flow
Training loss ↓
Validation loss ↓
 |
 v
Healthy learning

Training loss ↓
Validation loss ↑
 |
 v
Possible overfitting

36. LSTM Model

🐍 Python
from tensorflow.keras.layers import LSTM

lstm_model = Sequential([
 LSTM(
 64,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dense(1)
])

Compile:

🐍 Python
lstm_model.compile(
 optimizer="adam",
 loss="mse"
)

37. Train the LSTM

🐍 Python
lstm_history = lstm_model.fit(
 X_train,
 y_train,
 validation_data=(
 X_validation,
 y_validation
 ),
 epochs=30,
 batch_size=32,
 shuffle=False
)

38. GRU Model

🐍 Python
from tensorflow.keras.layers import GRU

gru_model = Sequential([
 GRU(
 64,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dense(1)
])

Compile:

🐍 Python
gru_model.compile(
 optimizer="adam",
 loss="mse"
)

Train:

🐍 Python
gru_history = gru_model.fit(
 X_train,
 y_train,
 validation_data=(
 X_validation,
 y_validation
 ),
 epochs=30,
 batch_size=32,
 shuffle=False
)

39. Early Stopping

Neural networks can overfit.

Early stopping monitors validation performance.

🐍 Python
from tensorflow.keras.callbacks import EarlyStopping

early_stopping = EarlyStopping(
 monitor="val_loss",
 patience=5,
 restore_best_weights=True
)

Use:

🐍 Python
lstm_history = lstm_model.fit(
 X_train,
 y_train,
 validation_data=(
 X_validation,
 y_validation
 ),
 epochs=100,
 batch_size=32,
 callbacks=[early_stopping],
 shuffle=False
)

40. Forecasting

Generate predictions:

🐍 Python
lstm_predictions_scaled = (
 lstm_model.predict(
 X_test
 )
)

Convert back to original units:

🐍 Python
lstm_predictions = scaler.inverse_transform(
 lstm_predictions_scaled
).ravel()

actual = scaler.inverse_transform(
 y_test.reshape(-1, 1)
).ravel()

41. Evaluate the Forecast

🐍 Python
from sklearn.metrics import (
 mean_absolute_error,
 mean_squared_error
)

mae = mean_absolute_error(
 actual,
 lstm_predictions
)

rmse = np.sqrt(
 mean_squared_error(
 actual,
 lstm_predictions
 )
)

print("MAE:", mae)
print("RMSE:", rmse)

Compare against the baselines from previous notebooks.

42. Visualize LSTM Forecast

🐍 Python
forecast_index = test.index[
 lookback:
]

plt.figure(figsize=(14, 5))

plt.plot(
 forecast_index,
 actual,
 label="Actual"
)

plt.plot(
 forecast_index,
 lstm_predictions,
 label="LSTM"
)

plt.title(
 "LSTM Time Series Forecast"
)

plt.xlabel("Date")
plt.ylabel("Sales")

plt.legend()

plt.show()

Visualization can reveal problems that a single metric cannot.

43. Compare RNN, LSTM, and GRU

Generate predictions for each:

🐍 Python
rnn_pred_scaled = rnn_model.predict(
 X_test
)

gru_pred_scaled = gru_model.predict(
 X_test
)

Inverse transform:

🐍 Python
rnn_pred = scaler.inverse_transform(
 rnn_pred_scaled
).ravel()

gru_pred = scaler.inverse_transform(
 gru_pred_scaled
).ravel()

Calculate metrics:

🐍 Python
results = pd.DataFrame({
 "model": [
 "RNN",
 "LSTM",
 "GRU"
 ],
 "MAE": [
 mean_absolute_error(
 actual,
 rnn_pred
 ),
 mean_absolute_error(
 actual,
 lstm_predictions
 ),
 mean_absolute_error(
 actual,
 gru_pred
 )
 ],
 "RMSE": [
 np.sqrt(
 mean_squared_error(
 actual,
 rnn_pred
 )
 ),
 np.sqrt(
 mean_squared_error(
 actual,
 lstm_predictions
 )
 ),
 np.sqrt(
 mean_squared_error(
 actual,
 gru_pred
 )
 )
 ]
})

results.sort_values("MAE")

44. Why LSTM/GRU May Not Always Win

It is tempting to assume:

text
Deep Learning
 >
XGBoost
 >
ARIMA

That is not generally true.

Performance depends on:

  • Dataset size
  • Signal-to-noise ratio
  • Forecast horizon
  • Seasonality
  • External variables
  • Feature engineering
  • Architecture
  • Hyperparameters
  • Validation methodology

For small structured datasets, classical or tree-based models can outperform deep learning.

45. Lookback Window

The lookback determines how much historical context the model receives.

Examples:

🐍 Python
lookback = 7

🐍 Python
lookback = 30

🐍 Python
lookback = 90

A larger window provides more context but also:

  • Increases computational cost.
  • Increases input size.
  • Can make optimization harder.
  • May introduce less relevant history.

The correct lookback is a modeling hyperparameter.

46. Selecting Lookback

Try multiple candidates:

text
7
14
30
60
90

Evaluate using the validation set.

Conceptually:

Architecture & Data Flow
Lookback
 |
 +--> 7 -> Validation MAE
 |
 +--> 14 -> Validation MAE
 |
 +--> 30 -> Validation MAE
 |
 +--> 60 -> Validation MAE

Choose based on validation performance and practical considerations.

47. Multivariate LSTM

Suppose the dataset contains:

text
sales
price
temperature
promotion

The input shape becomes:

›(samples, timesteps, 4)

Example model:

🐍 Python
multivariate_lstm = Sequential([
 LSTM(
 64,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dense(1)
])

multivariate_lstm.compile(
 optimizer="adam",
 loss="mse"
)

The network can learn relationships across variables and time.

48. Stacked LSTM

A deeper model can contain multiple recurrent layers.

🐍 Python
from tensorflow.keras.layers import Dropout

stacked_lstm = Sequential([
 LSTM(
 64,
 return_sequences=True,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dropout(0.2),

 LSTM(32),

 Dense(1)
])

Compile:

🐍 Python
stacked_lstm.compile(
 optimizer="adam",
 loss="mse"
)

The first LSTM returns a sequence so the second LSTM can process it.

49. Dropout

Dropout randomly disables a fraction of neural-network activations during training.

Example:

🐍 Python
Dropout(0.2)

This can help reduce overfitting.

But excessive dropout can hurt learning.

Use validation performance to determine whether regularization is helping.

50. Bidirectional RNNs for Forecasting

Bidirectional recurrent networks process sequences in both directions.

This can be useful in some sequence-processing tasks.

However, ordinary future forecasting requires special caution.

At prediction time, future observations are unavailable.

A bidirectional architecture must not be allowed to use information from future target timesteps.

For causal forecasting, standard forward temporal processing is generally the safer conceptual default.

51. One-Step Forecasting

One-step forecasting predicts:

y^t+1\hat{y}_{t+1}

from historical observations:

yt,yt−1,…y_t,y_{t-1},\ldots

This is the simplest forecasting setup.

The model predicts one future point at a time.

52. Multi-Step Forecasting

Suppose we want:

text
t+1
t+2
t+3
...
t+30

There are multiple strategies.

Recursive#

Predict one step, feed it back, and predict again.

Architecture & Data Flow
History -> t+1
 |
 v
 t+2
 |
 v
 t+3

Direct#

Train separate outputs/models for each horizon.

Multi-Output#

Predict the entire horizon simultaneously.

Each approach has trade-offs.

53. Recursive Deep Forecasting

A simplified pattern:

🐍 Python
def recursive_forecast(
 model,
 sequence,
 steps
):
 current_sequence = sequence.copy()
 predictions = []

 for _ in range(steps):
 prediction = model.predict(
 current_sequence[np.newaxis, ...],
 verbose=0
 )[0, 0]

 predictions.append(
 prediction
 )

 current_sequence = np.concatenate(
 [
 current_sequence[1:],
 [[prediction]]
 ],
 axis=0
 )

 return np.array(predictions)

This example assumes a single feature.

For multivariate forecasting, the update logic must account for how future feature values become available.

54. Why Recursive Forecasting Can Drift

Suppose:

Mathematical Formulation
True value = 100
Prediction = 103

The next prediction may use 103 as an input.

If the next prediction becomes:

›106

the error can propagate.

Therefore:

Architecture & Data Flow
Prediction error
 |
 v
Used as future input
 |
 v
New prediction error
 |
 v
Potential accumulation

This is known as error accumulation or exposure bias in certain sequence-learning settings.

55. Teacher Forcing Concept

During training, a sequence model may be given the true previous value.

During inference, it may have to use its own previous prediction.

This creates a mismatch:

text
Training:
True previous value

Inference:
Model prediction

This distinction becomes important for multi-step sequence forecasting.

56. Learning Rate

The optimizer's learning rate controls the magnitude of parameter updates.

Example:

🐍 Python
from tensorflow.keras.optimizers import Adam

optimizer = Adam(
 learning_rate=0.001
)

Too high:

›Training may become unstable.

Too low:

›Training may become extremely slow.

Learning rate is one of the most important neural-network hyperparameters.

57. Batch Size

Examples:

text
16
32
64
128

Smaller batches:

  • More frequent parameter updates.
  • Can introduce noisier gradients.

Larger batches:

  • More stable gradient estimates.
  • Can require more memory.

There is no universal best value.

58. Loss Functions

For regression forecasting, common losses include:

Mean Squared Error#

MSE=1n∑i=1n(yi−y^i)2MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i-\hat{y}_i)^2

Mean Absolute Error#

MAE=1n∑i=1n∣yi−y^i∣MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i-\hat{y}_i|

MSE penalizes large errors more heavily.

MAE is easier to interpret in the target's units.

59. Callbacks

Useful callbacks include:

🐍 Python
from tensorflow.keras.callbacks import (
 EarlyStopping,
 ReduceLROnPlateau
)

callbacks = [
 EarlyStopping(
 monitor="val_loss",
 patience=7,
 restore_best_weights=True
 ),
 ReduceLROnPlateau(
 monitor="val_loss",
 factor=0.5,
 patience=3
 )
]

These can help improve training stability.

60. Reproducibility

Set seeds where practical:

🐍 Python
import numpy as np
import tensorflow as tf

np.random.seed(42)
tf.random.set_seed(42)

Deep-learning training can still show some nondeterminism depending on:

  • Hardware
  • GPU operations
  • TensorFlow version
  • Parallel execution

Document your environment for reproducible experiments.

61. Model Capacity and Overfitting

A network with:

Architecture & Data Flow
Too few parameters
 |
 v
Underfitting

A network with:

Architecture & Data Flow
Too many parameters
 |
 v
Overfitting

Typical symptoms:

›Training loss keeps falling
Validation loss starts rising

Possible responses:

  • Reduce model size.
  • Add dropout.
  • Use early stopping.
  • Increase training data.
  • Improve features.
  • Reduce lookback complexity.
  • Tune learning rate.

62. Common Deep Learning Time-Series Mistakes

Mistake 1: Scaling Before Splitting#

This leaks information from validation/test data.

Mistake 2: Randomly Splitting Sequences#

Forecasting evaluation should respect temporal order.

Mistake 3: Creating Windows From Future Data#

Sequence construction must respect the forecast origin.

Mistake 4: Ignoring the Forecast Horizon#

One-step and 30-step forecasting are different problems.

Mistake 5: Using Future Covariates#

Only use future features that will genuinely be available.

Mistake 6: Comparing Only Training Loss#

Validation and test performance matter.

Mistake 7: Assuming LSTM Is Automatically Better#

Deep learning is not automatically superior to ARIMA or gradient-boosted trees.

Mistake 8: Using an Oversized Network#

More layers and units can increase overfitting.

63. Baselines Still Matter

Compare deep learning against:

text
Naive
Seasonal Naive
ARIMA/SARIMA
Random Forest
XGBoost
LightGBM
RNN
LSTM
GRU

The comparison should be based on the same:

  • Forecast horizon
  • Evaluation period
  • Target definition
  • Data availability
  • Metrics

A deep-learning model is useful only if its additional complexity is justified by improved performance or other practical benefits.

64. Model Evaluation

Use metrics such as:

🐍 Python
mae = mean_absolute_error(
 actual,
 predictions
)

rmse = np.sqrt(
 mean_squared_error(
 actual,
 predictions
 )
)

Later, Notebook 5 will cover:

  • MAE
  • RMSE
  • MAPE
  • sMAPE
  • Walk-forward validation

in greater detail.

65. Forecast Error Analysis

Create:

🐍 Python
error = (
 actual -
 lstm_predictions
)

Then inspect:

🐍 Python
error.mean()

and:

🐍 Python
pd.Series(error).describe()

Plot:

🐍 Python
plt.figure(figsize=(14, 5))

plt.plot(
 forecast_index,
 error
)

plt.axhline(
 0,
 linestyle="--"
)

plt.title(
 "LSTM Forecast Errors"
)

plt.show()

Look for:

  • Bias
  • Increasing error
  • Seasonal error
  • Large outliers
  • Periods where the model systematically fails

66. Deep Learning Development Workflow

A strong workflow is:

Architecture & Data Flow
Understand Data
 |
 v
Chronological Split
 |
 v
Fit Scaler on Training Only
 |
 v
Create Sequences
 |
 v
Naive Baseline
 |
 v
Simple RNN
 |
 v
LSTM
 |
 v
GRU
 |
 v
Tune Lookback / Architecture
 |
 v
Early Stopping
 |
 v
Evaluate on Test
 |
 v
Error Analysis

This prevents the common mistake of jumping directly into a large neural network.

67. Complete LSTM Template

🐍 Python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import (
 mean_absolute_error,
 mean_squared_error
)

import tensorflow as tf

from tensorflow.keras import Sequential

from tensorflow.keras.layers import (
 LSTM,
 Dense,
 Dropout
)

from tensorflow.keras.callbacks import (
 EarlyStopping
)

# --------------------------------------------------
# 1. Reproducibility
# --------------------------------------------------

np.random.seed(42)
tf.random.set_seed(42)

# --------------------------------------------------
# 2. Chronological split
# --------------------------------------------------

n = len(df)

train_end = int(n * 0.70)
validation_end = int(n * 0.85)

train = df.iloc[:train_end]
validation = df.iloc[
 train_end:validation_end
]
test = df.iloc[
 validation_end:
]

# --------------------------------------------------
# 3. Scale using training data only
# --------------------------------------------------

scaler = MinMaxScaler()

train_scaled = scaler.fit_transform(
 train[["sales"]]
)

validation_scaled = scaler.transform(
 validation[["sales"]]
)

test_scaled = scaler.transform(
 test[["sales"]]
)

# --------------------------------------------------
# 4. Create sequences
# --------------------------------------------------

lookback = 30

X_train, y_train = create_sequences(
 train_scaled,
 lookback
)

X_validation, y_validation = create_sequences(
 validation_scaled,
 lookback
)

X_test, y_test = create_sequences(
 test_scaled,
 lookback
)

# --------------------------------------------------
# 5. Build model
# --------------------------------------------------

model = Sequential([
 LSTM(
 64,
 input_shape=(
 X_train.shape[1],
 X_train.shape[2]
 )
 ),
 Dropout(0.2),
 Dense(1)
])

# --------------------------------------------------
# 6. Compile
# --------------------------------------------------

model.compile(
 optimizer="adam",
 loss="mse"
)

# --------------------------------------------------
# 7. Train
# --------------------------------------------------

early_stopping = EarlyStopping(
 monitor="val_loss",
 patience=7,
 restore_best_weights=True
)

history = model.fit(
 X_train,
 y_train,
 validation_data=(
 X_validation,
 y_validation
 ),
 epochs=100,
 batch_size=32,
 callbacks=[early_stopping],
 shuffle=False
)

# --------------------------------------------------
# 8. Predict
# --------------------------------------------------

predictions_scaled = model.predict(
 X_test
)

# --------------------------------------------------
# 9. Inverse transform
# --------------------------------------------------

predictions = scaler.inverse_transform(
 predictions_scaled
).ravel()

actual = scaler.inverse_transform(
 y_test.reshape(-1, 1)
).ravel()

# --------------------------------------------------
# 10. Evaluate
# --------------------------------------------------

mae = mean_absolute_error(
 actual,
 predictions
)

rmse = np.sqrt(
 mean_squared_error(
 actual,
 predictions
 )
)

print("MAE:", mae)
print("RMSE:", rmse)

# --------------------------------------------------
# 11. Visualize
# --------------------------------------------------

forecast_index = test.index[
 lookback:
]

plt.figure(figsize=(14, 5))

plt.plot(
 forecast_index,
 actual,
 label="Actual"
)

plt.plot(
 forecast_index,
 predictions,
 label="LSTM"
)

plt.legend()

plt.title(
 "LSTM Forecast"
)

plt.show()

68. Exercises

Exercise 1: Sequence Creation#

Given:

›[10, 20, 30, 40, 50, 60]

create sequences with:

Mathematical Formulation
lookback = 3

Write the resulting inputs and targets manually.

Exercise 2: Lookback Comparison#

Train an LSTM using:

text
7
14
30
60

Compare validation MAE and RMSE.

Explain which lookback performed best and why you think it did.

Exercise 3: RNN vs LSTM vs GRU#

Train:

text
Simple RNN
LSTM
GRU

Keep:

  • Dataset
  • Lookback
  • Train/validation/test split
  • Optimizer
  • Evaluation metric

consistent.

Compare results.

Exercise 4: Multivariate Forecasting#

Add one or more external variables such as:

text
price
promotion
temperature

Build a multivariate LSTM.

Compare against the univariate model.

Exercise 5: Overfitting#

Train an LSTM with:

›1 layer
2 layers

Plot training and validation loss.

Determine whether the deeper model overfits.

69. Mini Project: Deep Learning Demand Forecasting

Build a complete deep-learning forecasting project.

Part A: Data#

Choose a real-world time series dataset.

Examples:

  • Retail demand
  • Electricity consumption
  • Website traffic
  • Transportation demand
  • Sensor measurements

Part B: Preparation#

  • Parse timestamps.
  • Sort chronologically.
  • Handle missing values.
  • Identify frequency.
  • Split chronologically.

Part C: Sequence Design#

Test multiple:

›lookback windows

Part D: Models#

Train:

  • RNN
  • LSTM
  • GRU

Part E: Evaluation#

Compare:

  • Naive baseline
  • Seasonal naive
  • Best ML model from Notebook 3
  • RNN
  • LSTM
  • GRU

Use:

  • MAE
  • RMSE

Part F: Error Analysis#

Investigate:

  • Forecast bias
  • Seasonal errors
  • Peak-demand errors
  • Long-horizon behavior

Part G: Recommendation#

Explain:

  1. Which model performed best?
  2. Was deep learning actually better?
  3. How much additional complexity did it introduce?
  4. What features or architecture changes could improve it?
  5. What would you deploy in production and why?

70. Key Takeaways

The central transformation is:

Architecture & Data Flow
Time Series
 |
 v
Sequence Windows
 |
 v
3D Tensor
(samples, timesteps, features)
 |
 v
RNN / LSTM / GRU
 |
 v
Forecast

Remember:

  • Feed-forward networks do not inherently maintain temporal state.
  • RNNs introduce recurrent hidden states.
  • Basic RNNs can struggle with long-term dependencies.
  • LSTMs use gated memory mechanisms.
  • GRUs provide a simpler gated alternative.
  • Sequence windows determine how much historical context the model sees.
  • Deep-learning inputs commonly use (samples, timesteps, features).
  • Scaling must be fitted using training data only.
  • Forecasting splits must preserve temporal order.
  • One-step and multi-step forecasting are different problems.
  • Recursive forecasting can accumulate errors.
  • Deep learning is not automatically better than statistical or tree-based models.
  • Baselines remain essential.
  • Validation and error analysis are more important than simply increasing model size.

71. Preparation for Notebook 5

We now have three major forecasting approaches:

Architecture & Data Flow
Classical Statistics
 |
 +--> ARIMA / SARIMA
 |
 v
Machine Learning
 |
 +--> RF / XGBoost / LightGBM
 |
 v
Deep Learning
 |
 +--> RNN / LSTM / GRU

The final notebook will move into advanced forecasting and production strategies:

Architecture & Data Flow
Advanced Forecasting
 |
 +--> Transformers
 |
 +--> Prophet
 |
 +--> Advanced Evaluation
 |
 +--> Walk-Forward Validation
 |
 +--> Multi-Step Forecasting
 |
 +--> Rolling Retraining
 |
 +--> Deployment Strategy
 |
 +--> Monitoring

The central question will become:

How do we select, validate, deploy, and maintain a forecasting system in the real world?

Knowledge Checkpoint

Deep Learning for Time Series Checkpoint

Q1.What three gating mechanisms regulate information flow in a standard Long Short-Term Memory (LSTM) cell?
AForget Gate, Input Gate, Output Gate
BAttention Gate, Reset Gate, Kernel Gate
CEncoder Gate, Decoder Gate, Softmax Gate
DPooling Gate, Stride Gate, Padding Gate
Q2.How does a Gated Recurrent Unit (GRU) simplify the LSTM architecture?
AIt merges the cell state and hidden state, using only two gates: Reset Gate and Update Gate.
BIt removes all matrix multiplications.
CIt runs strictly in non-sequential order.
DIt requires 4x more parameters than LSTM.
Q3.Why do standard vanilla RNNs struggle with capturing long-range historical dependencies?
AVanishing and exploding gradients during backpropagation through time (BPTT) cause gradients to decay exponentially over long sequence lengths.
BRNNs cannot process floating point numbers.
CRNNs only train on the first 5 time steps.
DRNNs lack activation functions.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
Modern Machine Learning for Time Series: Feature Engineering & Tree-Based Forecasting
Next Lesson
Advanced Time Series Architectures & Production Forecasting