Intermediate

75–120 min read

#Time Series#AR#MA#ARMA#ARIMA#SARIMA#Stationarity#AIC#BIC#Residual Diagnostics

Classical Statistical Time Series Models: ARIMA & SARIMA

A practical and mathematical introduction to classical statistical forecasting models, from white noise and autoregression through ARIMA and SARIMA, with model selection and residual diagnostics.

Classical Statistical Time Series Models: ARIMA & SARIMA

1. Learning Objectives#

By the end of this notebook, you should be able to:

Explain the intuition behind white noise, AR, MA, ARMA, ARIMA, and SARIMA.
Understand why stationarity matters for classical time-series models.
Apply transformations and differencing appropriately.
Use ACF and PACF as diagnostic tools for model identification.
Understand AR and MA model parameters mathematically and intuitively.
Fit ARIMA and SARIMA models using Python.
Compare candidate models using AIC and BIC.
Diagnose residuals after fitting a model.
Understand the role of Ljung-Box testing.
Produce forecasts and prediction intervals.
Avoid common modeling mistakes and temporal leakage.
Understand when classical statistical models are appropriate and when other approaches may be better.

2. From EDA to Statistical Forecasting

In Notebook 1, we learned how to investigate a time series before modeling.

The general workflow was:

Architecture & Data Flow
Raw Time Series
 |
 v
Data Quality
 |
 v
Visualization
 |
 v
Trend / Seasonality
 |
 v
Stationarity
 |
 v
ACF / PACF
 |
 v
Statistical Modeling

Now we move into statistical forecasting.

The key idea behind classical time-series modeling is:

Use the historical behavior of a series to model its temporal dependence and generate forecasts.

3. Why Classical Statistical Models?

Before modern machine learning and deep learning became popular, forecasting was heavily based on statistical models.

These models remain important because they are:

Interpretable
Computationally efficient
Strong for many structured forecasting problems
Well studied mathematically
Useful with relatively small datasets
Valuable as baselines for modern models

A good forecasting practitioner should understand classical models even when using modern ML.

4. The Main Model Family

The progression in this notebook is:

Architecture & Data Flow
White Noise
 |
 v
AR
 |
 +------> MA
 |
 v
 ARMA
 |
 v
 ARIMA
 |
 v
 SARIMA

Each model adds a new capability.

5. White Noise

White noise is the simplest useful reference process.

We can represent it as:

Y_t = \epsilon_t

where:

E[\epsilon_t] = 0

and the errors are uncorrelated across time.

A common assumption is:

\epsilon_t \sim WN(0,\sigma^2)

The important intuition is:

The past does not provide predictable information about the future.

6. Why White Noise Matters

Suppose your model produces residuals:

e_t = y_t - \hat{y}_t

If residuals still contain strong temporal structure, the model may have missed something.

Ideally, after fitting a sufficiently good model, residuals should behave approximately like white noise.

Therefore:

Architecture & Data Flow
Good model
 |
 v
Predictable structure captured
 |
 v
Residuals ≈ unpredictable noise

White noise is therefore both:

A simple time-series model.
A target behavior for model residuals.

7. Generate White Noise


🐍 PythonInteractive WebAssembly
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)

white_noise = np.random.normal(
 loc=0,
 scale=1,
 size=500
)

wn = pd.Series(
 white_noise,
 index=pd.date_range(
 "2024-01-01",
 periods=500,
 freq="D"
 )
)

wn.plot(figsize=(14, 5))

plt.title("White Noise")
plt.xlabel("Date")
plt.ylabel("Value")

plt.show()

You should observe random fluctuations around a stable mean.

8. White Noise and ACF


🐍 PythonInteractive WebAssembly
from statsmodels.graphics.tsaplots import plot_acf

plot_acf(
 wn,
 lags=40
)

plt.show()

For ideal white noise, most autocorrelations should be close to zero.

Small correlations can occur simply because the sample is finite.

9. The Autoregressive Model

The autoregressive model assumes that the current value depends on previous values.

An AR(1) model is:

Y_t = c + \phi_1Y_{t-1} + \epsilon_t

An AR(2) model is:

Y_t = c + \phi_1Y_{t-1} + \phi_2Y_{t-2} + \epsilon_t

More generally, AR( $p$ ):

Y_t = c + \sum_{i=1}^{p} \phi_iY_{t-i} + \epsilon_t

where:

$p$ = number of autoregressive lags
$\phi_i$ = coefficient for lag $i$
$c$ = intercept
$\epsilon_t$ = random error

10. Intuition Behind AR

Imagine tomorrow's temperature depends partly on today's temperature.

A simplified model might be:

Temperature_t = 10 + 0.8Temperature_{t-1} + \epsilon_t

The coefficient $0.8$ indicates strong dependence on the previous observation.

AR models are therefore useful when:

The current value contains information from its own recent history.

11. Generate an AR Process


🐍 PythonInteractive WebAssembly
from statsmodels.tsa.arima_process import ArmaProcess

ar = np.array([1, -0.8])

ma = np.array([1])

process = ArmaProcess(ar, ma)

ar_data = process.generate_sample(
 nsample=500
)

ar_series = pd.Series(
 ar_data,
 index=pd.date_range(
 "2024-01-01",
 periods=500,
 freq="D"
 )
)

ar_series.plot(figsize=(14, 5))

plt.title("AR(1) Process")
plt.show()

12. AR Model and ACF


🐍 PythonInteractive WebAssembly
plot_acf(
 ar_series,
 lags=40
)

plt.show()

An AR process often has an ACF that gradually decays rather than stopping abruptly.

This behavior provides clues when identifying AR-type structure.

13. Moving Average Model

The Moving Average model is different from a moving average used for smoothing.

An MA model models the current observation using previous error terms.

MA(1):

Y_t = \mu + \epsilon_t + \theta_1\epsilon_{t-1}

MA( $q$ ):

Y_t = \mu + \epsilon_t + \sum_{i=1}^{q} \theta_i\epsilon_{t-i}

The important point:

AR uses past observations; MA uses past shocks/errors.

14. Intuition Behind MA

Imagine an unexpected event occurs today.

If that shock influences tomorrow's value, the model can represent that through an MA term.

For example:

Architecture & Data Flow
Unexpected event
 |
 v
Current error
 |
 v
Influences next observation

An MA model captures short-lived effects of previous shocks.

15. Generate an MA Process


🐍 PythonInteractive WebAssembly
ar = np.array([1])

ma = np.array([1, 0.7])

process = ArmaProcess(ar, ma)

ma_data = process.generate_sample(
 nsample=500
)

ma_series = pd.Series(
 ma_data,
 index=pd.date_range(
 "2024-01-01",
 periods=500,
 freq="D"
 )
)

ma_series.plot(figsize=(14, 5))

plt.title("MA(1) Process")
plt.show()

16. MA Model and ACF


🐍 PythonInteractive WebAssembly
plot_acf(
 ma_series,
 lags=40
)

plt.show()

For a pure MA( $q$ ) process, the theoretical ACF cuts off after lag $q$ .

This is one of the classical identification heuristics:

Architecture & Data Flow
ACF cuts off
 |
 v
Possible MA structure

Real data is noisier, so this should be treated as evidence rather than a rigid rule.

17. AR vs MA

Model	Uses	Typical ACF Behavior	Typical PACF Behavior
AR( $p$ )	Past observations	Gradual decay	Cutoff around p
MA( $q$ )	Past errors	Cutoff around q	Gradual decay

These patterns are useful for model identification.

However, ACF/PACF should not be used mechanically.

Domain knowledge, diagnostics, and model comparison are also important.

18. ARMA

ARMA combines:

Autoregressive terms
Moving-average terms

ARMA( $p,q$ ):

Y_t = c + \sum_{i=1}^{p}\phi_iY_{t-i} + \epsilon_t + \sum_{j=1}^{q}\theta_j\epsilon_{t-j}

It is appropriate for stationary series where both autoregressive and shock-dependent behavior are present.

19. Important Limitation of ARMA

ARMA assumes the modeled series is stationary.

Suppose we have:

Architecture & Data Flow
Strong trend
 |
 v
Non-stationary series
 |
 v
ARMA may not be appropriate directly

This leads to ARIMA.

20. ARIMA

ARIMA stands for:

AutoRegressive Integrated Moving Average

It combines:

AR: autoregressive behavior
I: integration/differencing
MA: moving-average behavior

Written as:

ARIMA(p,d,q)

where:

$p$ = AR order
$d$ = number of differences
$q$ = MA order

21. The "I" in ARIMA

Integration refers to differencing.

First difference:

\Delta Y_t = Y_t - Y_{t-1}

Second difference:

\Delta^2Y_t = \Delta Y_t-\Delta Y_{t-1}

The purpose is usually to transform a non-stationary series into one that can be modeled more effectively using ARMA-like dynamics.

22. Example of Differencing


🐍 PythonInteractive WebAssembly
df["sales_diff"] = df["sales"].diff()

Visualize:


🐍 PythonInteractive WebAssembly
df["sales_diff"].plot(figsize=(14, 5))

plt.title("First-Differenced Series")
plt.show()

Then test stationarity again:


🐍 PythonInteractive WebAssembly
from statsmodels.tsa.stattools import adfuller

result = adfuller(
 df["sales_diff"].dropna()
)

print("ADF statistic:", result[0])
print("p-value:", result[1])

23. Choosing the Differencing Order

The goal is not:

Difference as many times as possible.

The goal is:

Use the smallest amount of differencing needed to obtain an appropriate stationary representation.

Too much differencing can introduce unnecessary complexity and may create undesirable statistical behavior.

A practical process:

Architecture & Data Flow
Original series
 |
 v
Check stationarity
 |
 +---- stationary ----> d = 0
 |
 v
Difference once
 |
 v
Check again
 |
 +---- stationary ----> d = 1
 |
 v
Consider another difference

In practice, diagnostics and domain knowledge should guide the choice.

24. ARIMA Notation

Consider:

›ARIMA(2,1,1)

It means:

Mathematical Formulation
p = 2
d = 1
q = 1

So the model has:

Two AR terms
One order of differencing
One MA term

Another example:

›ARIMA(1,0,2)

is effectively an ARMA(1,2) model because no differencing is applied.

25. Forecasting with ARIMA

We will use statsmodels.


🐍 PythonInteractive WebAssembly
from statsmodels.tsa.arima.model import ARIMA

model = ARIMA(
 df["sales"],
 order=(1, 1, 1)
)

fitted_model = model.fit()

print(fitted_model.summary())

The exact order should not be chosen blindly.

The next sections explain how to reason about candidate orders.

26. Model Identification Using ACF and PACF

A traditional workflow is:

Architecture & Data Flow
1. Make the series stationary
 |
 v
2. Plot ACF
 |
 v
3. Plot PACF
 |
 v
4. Propose candidate p and q
 |
 v
5. Fit several models
 |
 v
6. Compare AIC/BIC
 |
 v
7. Diagnose residuals

This is better than selecting a model from one plot alone.

27. ACF Heuristics

For a stationary series:

Possible AR( $p$ )#

ACF may decay gradually.

Possible MA( $q$ )#

ACF may show a sharp cutoff after lag $q$ .

Seasonal Structure#

Large spikes may occur around seasonal lags.

For example:

text
Lag 7
Lag 14
Lag 21

may indicate weekly dependence in daily data.

28. PACF Heuristics

For a stationary series:

Possible AR( $p$ )#

PACF may show a cutoff around lag $p$ .

Possible MA( $q$ )#

PACF may decay gradually.

These are classical rules of thumb.

Modern workflows often combine these clues with automated search, information criteria, diagnostics, and validation.

29. AIC and BIC

When comparing models, we want a good balance between:

Model fit
Model complexity

AIC:

AIC = 2k - 2\log(L)

BIC:

BIC = k\log(n)-2\log(L)

where:

$k$ = number of estimated parameters
$L$ = likelihood
$n$ = number of observations

Lower values are generally preferred when comparing models fitted to the same data under comparable conditions.

30. Why Not Simply Choose the Lowest AIC?

Information criteria are useful, but they are not the final decision-maker.

A model with a lower AIC can still:

Have poor residual diagnostics
Generalize poorly
Be unstable
Be inappropriate for the forecasting objective

A stronger workflow is:

text
AIC/BIC
 +
Residual diagnostics
 +
Out-of-sample validation
 +
Domain knowledge

31. Candidate Model Search

Instead of guessing one model:


🐍 PythonInteractive WebAssembly
orders = [
 (1, 1, 0),
 (0, 1, 1),
 (1, 1, 1),
 (2, 1, 1),
 (1, 1, 2),
 (2, 1, 2)
]

Fit candidates:


🐍 PythonInteractive WebAssembly
results = []

for order in orders:
 model = ARIMA(
 df["sales"],
 order=order
 )

 fitted = model.fit()

 results.append({
 "order": order,
 "aic": fitted.aic,
 "bic": fitted.bic
 })

results_df = pd.DataFrame(results)

results_df.sort_values("aic")

This provides a systematic comparison.

32. Residual Diagnostics

After fitting a model, inspect its residuals.


🐍 PythonInteractive WebAssembly
residuals = fitted_model.resid

Plot:


🐍 PythonInteractive WebAssembly
residuals.plot(figsize=(14, 5))

plt.title("ARIMA Residuals")
plt.show()

Ask:

Is there remaining trend?
Is there seasonality?
Are there large outliers?
Is variance approximately stable?
Is there remaining autocorrelation?

33. Residual ACF


🐍 PythonInteractive WebAssembly
plot_acf(
 residuals.dropna(),
 lags=40
)

plt.show()

A good model should leave relatively little systematic autocorrelation.

If strong spikes remain, the model may have failed to capture temporal dependence.

34. Ljung-Box Test

The Ljung-Box test can evaluate whether a group of autocorrelations is jointly consistent with white noise.


🐍 PythonInteractive WebAssembly
from statsmodels.stats.diagnostic import acorr_ljungbox

lb_test = acorr_ljungbox(
 residuals.dropna(),
 lags=[10, 20],
 return_df=True
)

lb_test

A common interpretation:

Architecture & Data Flow
Small p-value
 |
 v
Evidence of remaining autocorrelation
 |
 v
Investigate the model

A larger p-value means there is insufficient evidence to reject the null hypothesis of no autocorrelation at the tested lags.

This is not proof that the residuals are perfectly white noise.

35. Model Diagnostics Summary

A reasonable fitted model should ideally have residuals that:

Architecture & Data Flow
Mean
 -> approximately zero

Variance
 -> reasonably stable

ACF
 -> little significant structure

Ljung-Box
 -> no strong evidence of remaining autocorrelation

Plot
 -> no obvious trend or seasonality

If these conditions are not met, revisit the model.

36. SARIMA

ARIMA handles non-seasonal temporal structure.

SARIMA extends ARIMA to seasonal patterns.

The notation is:

SARIMA(p,d,q)(P,D,Q)_s

where:

Non-seasonal terms#

$p$ = AR order
$d$ = differencing order
$q$ = MA order

Seasonal terms#

$P$ = seasonal AR order
$D$ = seasonal differencing order
$Q$ = seasonal MA order
$s$ = seasonal period

37. Example SARIMA Model

For daily data with weekly seasonality:

›SARIMA(1,1,1)(1,1,1,7)

For monthly data with yearly seasonality:

›SARIMA(1,1,1)(1,1,1,12)

The seasonal period must reflect the data frequency and business context.

38. Seasonal Differencing

Seasonal differencing compares observations separated by a seasonal period.

For seasonal period $s$ :

\Delta_sY_t = Y_t-Y_{t-s}

For weekly seasonality in daily data:

\Delta_7Y_t = Y_t-Y_{t-7}

In Python:


🐍 PythonInteractive WebAssembly
df["seasonal_diff"] = (
 df["sales"] -
 df["sales"].shift(7)
)

This can help remove repeating seasonal structure.

39. Fitting SARIMA

In statsmodels, SARIMA is implemented through the seasonal order of the ARIMA model.


🐍 PythonInteractive WebAssembly
from statsmodels.tsa.statespace.sarimax import SARIMAX

model = SARIMAX(
 df["sales"],
 order=(1, 1, 1),
 seasonal_order=(1, 1, 1, 7)
)

sarima_model = model.fit(
 disp=False
)

print(sarima_model.summary())

40. Forecasting

Suppose we want the next 30 observations:


🐍 PythonInteractive WebAssembly
forecast = sarima_model.get_forecast(
 steps=30
)

forecast_mean = forecast.predicted_mean

forecast_ci = forecast.conf_int()

Plot:


🐍 PythonInteractive WebAssembly
plt.figure(figsize=(14, 5))

plt.plot(
 df["sales"],
 label="Observed"
)

plt.plot(
 forecast_mean,
 label="Forecast"
)

plt.fill_between(
 forecast_ci.index,
 forecast_ci.iloc[:, 0],
 forecast_ci.iloc[:, 1],
 alpha=0.2
)

plt.title("SARIMA Forecast")
plt.legend()

plt.show()

41. Prediction Intervals

A forecast should not always be represented as one number.

For example:

Mathematical Formulation
Forecast = 250

95% prediction interval:
[220, 280]

The interval communicates uncertainty.

Generally:

Forecast uncertainty increases as the forecast horizon becomes longer.

This is important when communicating predictions to business stakeholders.

42. Train/Test Split for Time Series

Never randomly shuffle forecasting observations.

Use a chronological split:


🐍 PythonInteractive WebAssembly
split_date = "2025-06-01"

train = df.loc[
 df.index < split_date,
 "sales"
]

test = df.loc[
 df.index >= split_date,
 "sales"
]

Visualize:


🐍 PythonInteractive WebAssembly
plt.figure(figsize=(14, 5))

plt.plot(train, label="Train")
plt.plot(test, label="Test")

plt.legend()
plt.show()

The model learns only from the past.

43. Fit on Training Data


🐍 PythonInteractive WebAssembly
model = SARIMAX(
 train,
 order=(1, 1, 1),
 seasonal_order=(1, 1, 1, 7)
)

fitted = model.fit(
 disp=False
)

Forecast the test horizon:


🐍 PythonInteractive WebAssembly
forecast = fitted.get_forecast(
 steps=len(test)
)

predictions = forecast.predicted_mean

Then compare:


🐍 PythonInteractive WebAssembly
comparison = pd.DataFrame({
 "actual": test,
 "forecast": predictions
})

comparison.head()

44. Forecast Evaluation

Although detailed forecasting metrics are covered later, basic evaluation is useful here.


🐍 PythonInteractive WebAssembly
from sklearn.metrics import mean_absolute_error, mean_squared_error

mae = mean_absolute_error(
 test,
 predictions
)

rmse = np.sqrt(
 mean_squared_error(
 test,
 predictions
 )
)

print("MAE:", mae)
print("RMSE:", rmse)

Remember:

A low AIC does not guarantee the best out-of-sample forecasting performance.

This is why validation matters.

45. Baseline Forecasts

Always compare a complex model against a simple baseline.

Naive Forecast#

Predict the last observed value:

\hat{y}_{t+1}=y_t

Example:


🐍 PythonInteractive WebAssembly
naive_prediction = test.shift(1)

For a proper test setup, the first test prediction should use the final training observation.

Seasonal Naive#

For seasonal data:

\hat{y}_t = y_{t-s}

For weekly seasonality:


🐍 PythonInteractive WebAssembly
seasonal_naive = df["sales"].shift(7)

Baselines are essential because a sophisticated model should outperform a simple strategy to justify its complexity.

46. Practical Model Selection Workflow

A robust classical workflow:

Architecture & Data Flow
EDA
 |
 v
Identify frequency
 |
 v
Check trend and seasonality
 |
 v
Assess stationarity
 |
 v
Transform if necessary
 |
 v
Difference if necessary
 |
 v
Inspect ACF/PACF
 |
 v
Generate candidate models
 |
 v
Compare AIC/BIC
 |
 v
Check residuals
 |
 v
Evaluate on future holdout
 |
 v
Select final model

Do not skip the validation stage.

47. Transformations

Some time series have variance that grows with the level.

Example:

›Low level -> small fluctuations
High level -> large fluctuations

A logarithmic transformation can stabilize variance:

z_t = \log(y_t)

In Python:


🐍 PythonInteractive WebAssembly
df["log_sales"] = np.log(
 df["sales"]
)

For strictly positive data.

If zero values exist, alternatives such as:


🐍 PythonInteractive WebAssembly
np.log1p(df["sales"])

may be appropriate.

The transformation should be chosen based on the data-generating process, not automatically applied.

48. Back-Transforming Forecasts

If a model is trained on:

z_t=\log(y_t)

then a forecast on the log scale must be transformed back before communicating it in the original units.

Simple inverse transformation:


🐍 PythonInteractive WebAssembly
forecast_original = np.exp(
 forecast_log
)

However, for probabilistic forecasts, naïvely exponentiating the mean forecast can introduce bias.

This is an advanced issue to investigate when precise forecast calibration matters.

49. Over-Differencing

Over-differencing happens when a series is differenced more than necessary.

Potential consequences include:

Increased noise
Unnecessary model complexity
Poor forecasts
Artificial negative autocorrelation patterns

Always aim for the minimum appropriate differencing.

50. Under-Differencing

Under-differencing occurs when important non-stationary structure remains.

Symptoms can include:

Persistent trend
Slowly decaying ACF
Poor residual diagnostics
Unstable model behavior

The correct degree of differencing should be determined using multiple forms of evidence.

51. ARIMA vs SARIMA

Characteristic	ARIMA	SARIMA
AR terms	Yes	Yes
Differencing	Yes	Yes
MA terms	Yes	Yes
Seasonal AR	No	Yes
Seasonal differencing	No	Yes
Seasonal MA	No	Yes
Seasonal period	No	Yes
Typical use	Non-seasonal structure	Seasonal + non-seasonal structure

52. When Classical Models Work Well

ARIMA/SARIMA can be strong choices when:

The dataset is relatively small.
Temporal structure is clear.
Seasonality is well defined.
Historical behavior is informative.
Interpretability is important.
A fast baseline is required.

They are often excellent starting points.

53. When Classical Models May Struggle

Classical models can become challenging when:

There are many external predictors.
Multiple complex seasonalities exist.
Relationships are strongly nonlinear.
Structural behavior changes frequently.
There are very large datasets.
Complex interactions exist across many variables.

This is where machine learning and deep learning can become attractive.

That does not mean modern models automatically perform better.

The correct approach is empirical comparison.

54. Common Beginner Mistakes

Mistake 1: Treating AIC as the Final Answer#

AIC helps compare models but does not replace validation.

Mistake 2: Randomly Splitting the Dataset#

Forecasting must respect temporal ordering.

Mistake 3: Blindly Differencing#

More differencing is not automatically better.

Mistake 4: Ignoring Seasonality#

A strong seasonal pattern may require SARIMA rather than plain ARIMA.

Mistake 5: Ignoring Residuals#

A model summary alone is not enough.

Mistake 6: Using ACF/PACF Mechanically#

They provide clues, not guaranteed model orders.

Mistake 7: Using Too Many Parameters#

Highly complex models can overfit historical behavior.

Mistake 8: Ignoring Baselines#

A complex model should be compared against simple forecasting strategies.

55. Complete Classical Forecasting Template


🐍 PythonInteractive WebAssembly
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from statsmodels.stats.diagnostic import acorr_ljungbox

from sklearn.metrics import (
 mean_absolute_error,
 mean_squared_error
)

# --------------------------------------------------
# 1. Load and prepare data
# --------------------------------------------------

df = pd.read_csv("time_series.csv")

df["date"] = pd.to_datetime(df["date"])

df = (
 df
 .sort_values("date")
 .set_index("date")
)

# --------------------------------------------------
# 2. Chronological split
# --------------------------------------------------

split_date = "2025-06-01"

train = df.loc[
 df.index < split_date,
 "target"
]

test = df.loc[
 df.index >= split_date,
 "target"
]

# --------------------------------------------------
# 3. Fit SARIMA
# --------------------------------------------------

model = SARIMAX(
 train,
 order=(1, 1, 1),
 seasonal_order=(1, 1, 1, 7)
)

fitted = model.fit(
 disp=False
)

# --------------------------------------------------
# 4. Forecast
# --------------------------------------------------

forecast_result = fitted.get_forecast(
 steps=len(test)
)

predictions = forecast_result.predicted_mean

confidence_intervals = (
 forecast_result.conf_int()
)

# --------------------------------------------------
# 5. Evaluate
# --------------------------------------------------

mae = mean_absolute_error(
 test,
 predictions
)

rmse = np.sqrt(
 mean_squared_error(
 test,
 predictions
 )
)

print("MAE:", mae)
print("RMSE:", rmse)

# --------------------------------------------------
# 6. Residual diagnostics
# --------------------------------------------------

residuals = fitted.resid

plot_acf(
 residuals.dropna(),
 lags=40
)

plt.show()

print(
 acorr_ljungbox(
 residuals.dropna(),
 lags=[10, 20],
 return_df=True
 )
)

# --------------------------------------------------
# 7. Forecast visualization
# --------------------------------------------------

plt.figure(figsize=(14, 5))

plt.plot(
 train,
 label="Train"
)

plt.plot(
 test,
 label="Actual"
)

plt.plot(
 predictions,
 label="Forecast"
)

plt.fill_between(
 confidence_intervals.index,
 confidence_intervals.iloc[:, 0],
 confidence_intervals.iloc[:, 1],
 alpha=0.2
)

plt.title("SARIMA Forecast")
plt.legend()

plt.show()

56. Exercises

Exercise 1: White Noise#

Generate a white-noise series.
Plot it.
Plot its ACF.
Explain why the ACF behaves differently from an AR process.

Exercise 2: AR Process#

Generate an AR(1) process with:

Mathematical Formulation
phi = 0.3
phi = 0.8
phi = -0.5

Compare the resulting behavior.

Explain how changing the coefficient affects temporal dependence.

Exercise 3: Differencing#

Using the sales dataset:

Plot the original series.
Run the ADF test.
Apply first differencing.
Run ADF again.
Compare ACF before and after differencing.

Exercise 4: ARIMA Model Search#

Evaluate several candidate models:

text
ARIMA(1,1,0)
ARIMA(0,1,1)
ARIMA(1,1,1)
ARIMA(2,1,1)
ARIMA(1,1,2)
ARIMA(2,1,2)

Create a table containing:

Order
AIC
BIC

Then investigate the residuals of the best candidates.

Exercise 5: SARIMA#

For daily data with weekly seasonality:

Mathematical Formulation
seasonal period = 7

Compare:

›ARIMA(1,1,1)

against:

›SARIMA(1,1,1)(1,1,1,7)

Evaluate both models on a chronological holdout set.

Do not select the model solely using AIC.

57. Mini Project: Classical Forecasting

Choose a real-world dataset with a meaningful time component.

Examples:

Retail sales
Electricity demand
Website traffic
Product orders
Transportation demand

Complete the following:

Part A: EDA#

Identify frequency.
Visualize the series.
Identify trend.
Identify seasonality.
Inspect ACF/PACF.

Part B: Stationarity#

Run ADF.
Apply transformation if appropriate.
Apply differencing if appropriate.
Explain the reasoning.

Part C: Model Candidates#

Train several:

ARIMA models.
SARIMA models where appropriate.

Part D: Model Selection#

Compare:

AIC
BIC
Residual diagnostics
Holdout forecasting performance

Part E: Forecast#

Produce:

Point forecasts.
Prediction intervals.
Forecast visualization.

Part F: Interpretation#

Answer:

What temporal structure did the model capture?
Was seasonality important?
Were residuals approximately uncorrelated?
Did the model outperform a naive baseline?
What limitations remain?

58. Key Takeaways

The main progression is:

Architecture & Data Flow
White Noise
 |
 v
AR
 |
 +---- MA
 |
 v
 ARMA
 |
 v
 ARIMA
 |
 v
 SARIMA

Remember:

AR models use past observations.
MA models use past errors.
ARMA combines AR and MA for stationary series.
ARIMA adds differencing.
SARIMA adds seasonal structure.
ACF and PACF help identify candidate structures.
AIC and BIC help compare model complexity and fit.
Residual diagnostics tell us what the model failed to explain.
Forecasting validation must respect time order.
A strong baseline is essential.
The best model is not necessarily the most complicated model.

59. Preparation for Notebook 3

We have now learned how statistical models directly represent temporal dependence.

The next question is:

What if we treat time-series forecasting as a supervised machine-learning problem?

The progression will be:

Architecture & Data Flow
Time Series
 |
 v
Create Lag Features
 |
 v
Create Rolling Features
 |
 v
Create Calendar Features
 |
 v
Supervised ML Dataset
 |
 v
Random Forest
 |
 v
XGBoost
 |
 v
LightGBM
 |
 v
Multiple Seasonalities
 |
 v
Statistical vs ML Comparison

Notebook 3 will focus on feature engineering for time series and tree-based machine learning.

The critical lesson will be:

We can convert temporal structure into features that standard supervised-learning algorithms can learn from, but we must do this without leaking future information.

Knowledge Checkpoint

Classical Models: ARIMA & SARIMA Checkpoint

Q1.What do the three parameters $(p, d, q)$ represent in an $\text{ARIMA}(p, d, q)$ model?

A$p$: Autoregressive order, $d$: Degree of differencing, $q$: Moving Average order

B$p$: Polynomial degree, $d$: Decay rate, $q$: Quantile

C$p$: Periodicity, $d$: Drift, $q$: Quality factor

D$p$: Precision, $d$: Dimension, $q$: Queue length

Q2.How does Seasonal ARIMA (SARIMA) extend standard ARIMA?

ABy adding seasonal autoregressive $(P)$, seasonal differencing $(D)$, and seasonal moving average $(Q)$ terms at seasonal lag period $s$: $\text{SARIMA}(p,d,q)(P,D,Q)_s$.

BBy training on seasonal weather APIs.

CBy discarding all non-seasonal data.

DBy running ARIMA exclusively in summer months.

Q3.What information criterion is standard for penalizing complexity when selecting optimal ARIMA orders?

AAkaike Information Criterion (AIC) / Bayesian Information Criterion (BIC)

BMean Absolute Percentage Error (MAPE)

CGini Impurity

DF1-Score

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Time Series Foundations & Exploratory Analysis

Next Lesson

Modern Machine Learning for Time Series: Feature Engineering & Tree-Based Forecasting