∇GradientSchool
Advanced
90–150 min read
#Time Series#Feature Engineering#Lag Features#Rolling Statistics#Random Forest#XGBoost#LightGBM#Forecasting#Data Leakage

Modern Machine Learning for Time Series: Feature Engineering & Tree-Based Forecasting

Learn how to transform time series into supervised learning problems, engineer leakage-safe temporal features, train tree-based models, handle multiple seasonalities, and compare machine learning forecasting with classical statistical approaches.

Modern Machine Learning for Time Series: Feature Engineering & Tree-Based Forecasting

1. Learning Objectives#

By the end of this notebook, you should be able to:

  • Explain how time series forecasting can be formulated as supervised learning.
  • Understand why ordinary tabular ML models cannot automatically understand temporal order.
  • Create lag, rolling, expanding, calendar, and seasonal features.
  • Build forecasting datasets for one-step and multi-step prediction.
  • Prevent future-data leakage during feature engineering.
  • Perform chronological train/validation/test splitting.
  • Build strong baseline models.
  • Train Random Forest, XGBoost, and LightGBM forecasting models.
  • Understand the strengths and limitations of tree-based models for time series.
  • Handle multiple seasonal patterns using engineered features.
  • Compare ML forecasts against classical statistical baselines.
  • Interpret feature importance in a time-series setting.
  • Understand recursive versus direct forecasting strategies.
  • Build a reusable time-series ML pipeline.

2. From Statistical Forecasting to Machine Learning

In Notebook 2, we modeled temporal structure directly using:

text
AR
MA
ARMA
ARIMA
SARIMA

Those models explicitly represent relationships between observations over time.

Now we take a different approach:

Convert temporal relationships into features and let a supervised machine-learning model learn the relationship.

The conceptual transformation is:

Architecture & Data Flow
Original Time Series

Date Sales
Jan 01 120
Jan 02 135
Jan 03 128
Jan 04 145
 |
 v
Feature Engineering
 |
 v
Lag / Rolling / Calendar Features
 |
 v
Supervised Learning Dataset
 |
 v
ML Model
 |
 v
Forecast

This approach allows models such as Random Forest, XGBoost, and LightGBM to work with time-series data.

3. Why Feature Engineering Is Necessary

A normal tree-based model does not inherently know that:

›2025-01-10

comes immediately after:

›2025-01-09

It also does not automatically understand that:

›Yesterday's sales

may be useful for predicting:

›Today's sales

We therefore explicitly create features such as:

text
sales_lag_1
sales_lag_7
rolling_mean_7
rolling_std_7
day_of_week
month
is_weekend

The model can then learn relationships such as:

yt=f(yt−1,yt−7,rollingMeant,dayOfWeekt,montht,…)y_t = f( y_{t-1}, y_{t-7}, rollingMean_t, dayOfWeek_t, month_t, \ldots )

4. The Core Transformation

Suppose the original data is:

DateSales
Jan 1100
Jan 2110
Jan 3105
Jan 4120
Jan 5125

Create a one-lag feature:

DateLag 1Target
Jan 2100110
Jan 3110105
Jan 4105120
Jan 5120125

Now this is a standard supervised-learning problem:

Xt=yt−1X_t = y_{t-1} yt=targety_t = target

The model learns:

y^t=f(yt−1)\hat{y}_t = f(y_{t-1})

5. Why This Is Powerful

Once the series is represented as features, we can include much more information.

For example:

text
Previous day sales
Previous week sales
Previous month sales
7-day average
30-day average
7-day volatility
Day of week
Month
Holiday indicator
Promotion indicator
Price
Weather

Then:

y^t=f(LagFeatures,RollingFeatures,CalendarFeatures,ExternalFeatures)\hat{y}_t = f( LagFeatures, RollingFeatures, CalendarFeatures, ExternalFeatures )

This is where machine learning becomes particularly useful.

6. Dataset Setup

We will use a synthetic retail-sales dataset so the notebook is reproducible.

🐍 Python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)

dates = pd.date_range(
 start="2022-01-01",
 periods=3 * 365,
 freq="D"
)

n = len(dates)

trend = np.linspace(100, 180, n)

weekly = (
 15 *
 np.sin(
 2 * np.pi * np.arange(n) / 7
 )
)

yearly = (
 10 *
 np.sin(
 2 * np.pi * np.arange(n) / 365
 )
)

noise = np.random.normal(
 0,
 5,
 n
)

sales = (
 trend
 + weekly
 + yearly
 + noise
)

df = pd.DataFrame({
 "date": dates,
 "sales": sales
})

df.head()

7. Prepare the Time Index

🐍 Python
df["date"] = pd.to_datetime(df["date"])

df = (
 df
 .sort_values("date")
 .set_index("date")
)

df.head()

Always sort chronologically before feature engineering.

8. Train, Validation, and Test Sets

For forecasting, use chronological splits.

Example:

Architecture & Data Flow
2022 ---------------- 2023 ---------------- 2024
|---------------------|----------------------|
 Train Validation/Test

A better structure is:

Architecture & Data Flow
Training Validation Test
|---------------|------------------|
Past Later Future

For example:

🐍 Python
train_end = "2024-01-01"
validation_end = "2024-07-01"

train = df.loc[
 df.index < train_end
]

validation = df.loc[
 (df.index >= train_end) &
 (df.index < validation_end)
]

test = df.loc[
 df.index >= validation_end
]

The exact dates should be adapted to the dataset.

9. Why Random Splitting Is Dangerous

Avoid:

🐍 Python
from sklearn.model_selection import train_test_split

train_test_split(
 df,
 test_size=0.2,
 random_state=42
)

for ordinary forecasting.

Random splitting can create:

text
Training:
Jan 1
Jan 3
Jan 5
Jan 8
Jan 10

Test:
Jan 2
Jan 4
Jan 6
Jan 7

The model is effectively learning from information that occurs after some test observations.

This is a form of temporal leakage.

10. Lag Features

Lag features are among the most important time-series ML features.

A lag of one:

Lag1(t)=yt−1Lag_1(t) = y_{t-1}

A lag of seven:

Lag7(t)=yt−7Lag_7(t) = y_{t-7}

In pandas:

🐍 Python
df["lag_1"] = df["sales"].shift(1)

df["lag_7"] = df["sales"].shift(7)

df["lag_14"] = df["sales"].shift(14)

df["lag_28"] = df["sales"].shift(28)

Inspect:

🐍 Python
df.head(35)

The first rows contain missing values because historical observations do not exist for those lags.

11. Why Different Lags Matter

Different lags represent different types of memory.

Architecture & Data Flow
lag_1
 -> recent behavior

lag_7
 -> weekly behavior

lag_14
 -> two-week behavior

lag_28
 -> approximately monthly behavior

The correct lags depend on the problem.

For hourly data:

text
lag_1
lag_24
lag_168

might represent:

  • Previous hour
  • Previous day
  • Previous week

12. Seasonal Lag Features

If a dataset has strong seasonality, seasonal lags can be extremely useful.

For daily retail data:

🐍 Python
df["lag_7"] = df["sales"].shift(7)

For monthly data:

🐍 Python
df["lag_12"] = df["sales"].shift(12)

For hourly data:

🐍 Python
df["lag_24"] = df["sales"].shift(24)
df["lag_168"] = df["sales"].shift(168)

These features provide direct access to historical seasonal behavior.

13. Rolling Features

Rolling statistics summarize recent history.

For example:

RollingMeant=Mean(yt−1,yt−2,…,yt−k)RollingMean_t = Mean(y_{t-1}, y_{t-2}, \ldots, y_{t-k})

A common implementation:

🐍 Python
df["rolling_mean_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .mean()
)

Notice the:

🐍 Python
.shift(1)

This is important.

14. Why Shift Before Rolling?

Suppose we want to predict:

›Sales on January 10

We must not calculate a feature using:

›Sales on January 10

because that is the target itself.

Instead, use historical values:

text
January 9
January 8
January 7
...

Therefore:

🐍 Python
df["rolling_mean_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .mean()
)

is safer for one-step-ahead forecasting than:

🐍 Python
df["sales"].rolling(7).mean()

when the current target is included in the window.

This is one of the most important leakage-prevention rules in this notebook.

15. Rolling Standard Deviation

Rolling standard deviation captures recent variability.

🐍 Python
df["rolling_std_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .std()
)

This can help a model understand:

text
Stable period
vs.
High-volatility period

16. Multiple Rolling Windows

🐍 Python
for window in [7, 14, 28]:
 df[f"rolling_mean_{window}"] = (
 df["sales"]
 .shift(1)
 .rolling(window)
 .mean()
 )

 df[f"rolling_std_{window}"] = (
 df["sales"]
 .shift(1)
 .rolling(window)
 .std()
 )

This gives the model multiple temporal scales.

17. Expanding Features

An expanding statistic uses all available historical observations.

For example:

🐍 Python
df["expanding_mean"] = (
 df["sales"]
 .shift(1)
 .expanding()
 .mean()
)

This represents the historical average up to the previous observation.

Expanding features can be useful for:

  • Long-term averages
  • Cumulative statistics
  • Historical baselines

But they should still be calculated using only information available before the prediction timestamp.

18. Calendar Features

Calendar information often provides strong predictive signals.

🐍 Python
df["day_of_week"] = df.index.dayofweek

df["day_of_month"] = df.index.day

df["week_of_year"] = (
 df.index.isocalendar().week.astype(int)
)

df["month"] = df.index.month

df["quarter"] = df.index.quarter

df["year"] = df.index.year

Weekend indicator:

🐍 Python
df["is_weekend"] = (
 df["day_of_week"] >= 5
).astype(int)

19. Cyclical Encoding

Calendar variables are cyclical.

For example:

›Sunday -> Monday

is not naturally represented by:

›6 -> 0

A model may interpret 6 and 0 as far apart.

We can encode periodic variables using sine and cosine.

For day of week:

sinDay=sin(2πdayOfWeek7)sinDay = sin\left( 2\pi\frac{dayOfWeek}{7} \right) cosDay=cos(2πdayOfWeek7)cosDay = cos\left( 2\pi\frac{dayOfWeek}{7} \right)

Python:

🐍 Python
df["dow_sin"] = np.sin(
 2 * np.pi * df["day_of_week"] / 7
)

df["dow_cos"] = np.cos(
 2 * np.pi * df["day_of_week"] / 7
)

This creates a continuous representation of the cycle.

20. Monthly Cyclical Features

🐍 Python
df["month_sin"] = np.sin(
 2 * np.pi * df["month"] / 12
)

df["month_cos"] = np.cos(
 2 * np.pi * df["month"] / 12
)

Cyclical encoding is particularly useful for models that benefit from smooth numerical representations.

Tree models can sometimes learn categorical calendar effects directly, but cyclical features can still be useful depending on the problem.

21. External Variables

Time-series forecasting does not have to rely only on the target's history.

External variables may include:

  • Price
  • Promotions
  • Weather
  • Holidays
  • Advertising spend
  • Economic indicators
  • Inventory
  • Competitor pricing

Example:

Salest=f(Salest−1,Salest−7,Pricet,Promotiont,Temperaturet)Sales_t = f( Sales_{t-1}, Sales_{t-7}, Price_t, Promotion_t, Temperature_t )

These are often called exogenous or external features.

22. The Most Important Leakage Question

For every feature, ask:

Would this value actually be known at the exact time the forecast is generated?

For example:

Safe#

›Yesterday's sales
Known at prediction time

Potentially unsafe#

›Tomorrow's realized temperature

unless the forecasting system genuinely has a future weather forecast available and that forecast is what will be supplied to the model.

Unsafe#

›Future actual sales

because the target itself is unknown.

23. Building a Feature Engineering Function

Create reusable code.

🐍 Python
def create_time_features(df, target):
 data = df.copy()

 # Lag features
 for lag in [1, 7, 14, 28]:
 data[f"lag_{lag}"] = (
 data[target].shift(lag)
 )

 # Rolling features
 for window in [7, 14, 28]:
 data[f"rolling_mean_{window}"] = (
 data[target]
 .shift(1)
 .rolling(window)
 .mean()
 )

 data[f"rolling_std_{window}"] = (
 data[target]
 .shift(1)
 .rolling(window)
 .std()
 )

 # Calendar features
 data["day_of_week"] = data.index.dayofweek
 data["month"] = data.index.month
 data["quarter"] = data.index.quarter
 data["year"] = data.index.year

 data["is_weekend"] = (
 data["day_of_week"] >= 5
 ).astype(int)

 # Cyclical encoding
 data["dow_sin"] = np.sin(
 2 * np.pi * data["day_of_week"] / 7
 )

 data["dow_cos"] = np.cos(
 2 * np.pi * data["day_of_week"] / 7
 )

 data["month_sin"] = np.sin(
 2 * np.pi * data["month"] / 12
 )

 data["month_cos"] = np.cos(
 2 * np.pi * data["month"] / 12
 )

 return data

24. Create the ML Dataset

🐍 Python
model_df = create_time_features(
 df,
 target="sales"
)

model_df.head(35)

Remove rows where lag features are unavailable:

🐍 Python
model_df = model_df.dropna()

Now we have a supervised-learning dataset.

25. Define Features and Target

🐍 Python
target = "sales"

features = [
 "lag_1",
 "lag_7",
 "lag_14",
 "lag_28",
 "rolling_mean_7",
 "rolling_mean_14",
 "rolling_mean_28",
 "rolling_std_7",
 "rolling_std_14",
 "rolling_std_28",
 "day_of_week",
 "month",
 "quarter",
 "year",
 "is_weekend",
 "dow_sin",
 "dow_cos",
 "month_sin",
 "month_cos"
]

X = model_df[features]
y = model_df[target]

26. Chronological Dataset Splitting

🐍 Python
train_end = "2024-01-01"
validation_end = "2024-07-01"

X_train = X.loc[
 X.index < train_end
]

y_train = y.loc[
 y.index < train_end
]

X_validation = X.loc[
 (X.index >= train_end) &
 (X.index < validation_end)
]

y_validation = y.loc[
 (y.index >= train_end) &
 (y.index < validation_end)
]

X_test = X.loc[
 X.index >= validation_end
]

y_test = y.loc[
 y.index >= validation_end
]

The ordering remains intact.

27. Baseline First

Before training a complex ML model, establish a baseline.

Naive Forecast#

For one-step forecasting:

y^t=yt−1\hat{y}_t = y_{t-1} 
🐍 Python
naive_predictions = (
 model_df["lag_1"]
 .loc[y_test.index]
)

Evaluate:

🐍 Python
from sklearn.metrics import mean_absolute_error

naive_mae = mean_absolute_error(
 y_test,
 naive_predictions
)

print("Naive MAE:", naive_mae)

The ML model should beat this baseline.

28. Random Forest

Random Forest is an ensemble of decision trees.

Conceptually:

Architecture & Data Flow
Dataset
 |
 +--> Tree 1
 |
 +--> Tree 2
 |
 +--> Tree 3
 |
 +--> ...
 |
 v
Average predictions

Random Forest can learn nonlinear relationships between temporal features.

29. Train Random Forest

🐍 Python
from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(
 n_estimators=300,
 max_depth=None,
 min_samples_leaf=2,
 random_state=42,
 n_jobs=-1
)

rf.fit(
 X_train,
 y_train
)

Predict:

🐍 Python
rf_validation_predictions = rf.predict(
 X_validation
)

Evaluate:

🐍 Python
rf_validation_mae = mean_absolute_error(
 y_validation,
 rf_validation_predictions
)

print(
 "Random Forest validation MAE:",
 rf_validation_mae
)

30. Random Forest: Strengths

Advantages:

  • Handles nonlinear relationships.
  • Requires little preprocessing.
  • Robust to many feature types.
  • Captures feature interactions.
  • Useful as a strong baseline.

Limitations:

  • Can become large.
  • May not extrapolate trends well.
  • Depends heavily on feature engineering.
  • Does not inherently understand temporal order.

The last point is critical.

Random Forest does not "know time."

It learns from the features we provide.

31. XGBoost

XGBoost is a gradient-boosted tree algorithm.

Instead of training independent trees and averaging them, boosting builds trees sequentially.

Conceptually:

Architecture & Data Flow
Tree 1
 |
 v
Errors
 |
 v
Tree 2 learns from errors
 |
 v
Errors
 |
 v
Tree 3
 |
 v
Final ensemble

This can produce highly accurate predictions on structured tabular data.

32. Install XGBoost

In an environment where XGBoost is not installed:

bash
pip install xgboost

Import:

🐍 Python
from xgboost import XGBRegressor

33. Train XGBoost

🐍 Python
xgb = XGBRegressor(
 n_estimators=500,
 learning_rate=0.05,
 max_depth=6,
 subsample=0.8,
 colsample_bytree=0.8,
 objective="reg:squarederror",
 random_state=42
)

xgb.fit(
 X_train,
 y_train
)

Predict:

🐍 Python
xgb_validation_predictions = xgb.predict(
 X_validation
)

Evaluate:

🐍 Python
xgb_validation_mae = mean_absolute_error(
 y_validation,
 xgb_validation_predictions
)

print(
 "XGBoost validation MAE:",
 xgb_validation_mae
)

34. XGBoost and Time Series

XGBoost can learn relationships such as:

text
If:
 lag_7 is high
 AND
 day_of_week is Saturday
 AND
 rolling_mean_28 is increasing

Then:
 predicted sales may be high

This nonlinear interaction is difficult to represent with simple linear models.

35. LightGBM

LightGBM is another gradient-boosting framework designed for efficient tree-based learning.

Install if needed:

bash
pip install lightgbm

Import:

🐍 Python
from lightgbm import LGBMRegressor

36. Train LightGBM

🐍 Python
lgbm = LGBMRegressor(
 n_estimators=500,
 learning_rate=0.05,
 max_depth=-1,
 num_leaves=31,
 subsample=0.8,
 colsample_bytree=0.8,
 random_state=42
)

lgbm.fit(
 X_train,
 y_train
)

Predict:

🐍 Python
lgbm_validation_predictions = lgbm.predict(
 X_validation
)

Evaluate:

🐍 Python
lgbm_validation_mae = mean_absolute_error(
 y_validation,
 lgbm_validation_predictions
)

print(
 "LightGBM validation MAE:",
 lgbm_validation_mae
)

37. Compare Models

Create a simple comparison:

🐍 Python
comparison = pd.DataFrame({
 "model": [
 "Naive",
 "Random Forest",
 "XGBoost",
 "LightGBM"
 ],
 "validation_mae": [
 naive_mae,
 rf_validation_mae,
 xgb_validation_mae,
 lgbm_validation_mae
 ]
})

comparison.sort_values(
 "validation_mae"
)

The lower MAE is better.

But do not choose a model using one metric and one split alone for a serious production system.

38. Feature Importance

Tree models can provide feature importance.

For Random Forest:

🐍 Python
rf_importance = pd.Series(
 rf.feature_importances_,
 index=features
).sort_values(
 ascending=False
)

rf_importance

Visualize:

🐍 Python
rf_importance.head(15).sort_values().plot(
 kind="barh",
 figsize=(10, 6)
)

plt.title("Random Forest Feature Importance")
plt.xlabel("Importance")

plt.show()

39. XGBoost Feature Importance

🐍 Python
xgb_importance = pd.Series(
 xgb.feature_importances_,
 index=features
).sort_values(
 ascending=False
)

xgb_importance.head(15)

You may discover that:

text
lag_7
rolling_mean_7
lag_1
day_of_week

are highly important.

This provides useful insight into what temporal signals the model relies on.

40. Important Warning About Feature Importance

Feature importance is not the same as causality.

If:

›lag_7

is highly important, that means it helps prediction.

It does not mean:

The previous week's value causes the current value.

Feature importance tells us about predictive utility, not necessarily causal relationships.

41. Recursive Forecasting

A major challenge appears when forecasting multiple future periods.

Suppose we want:

text
Tomorrow
Day +2
Day +3
...
Day +30

For Day +1, we know historical lags.

But for Day +2:

Mathematical Formulation
lag_1 = predicted Day +1

Now the model uses its own prediction as an input.

This is recursive forecasting.

42. Recursive Forecasting Concept

Architecture & Data Flow
Historical data
 |
 v
Predict t+1
 |
 v
Use prediction as feature
 |
 v
Predict t+2
 |
 v
Use prediction as feature
 |
 v
Predict t+3
 |
 ...

This can cause forecast errors to accumulate.

43. Direct Multi-Step Forecasting

Another strategy is to train separate models for different horizons.

For example:

Architecture & Data Flow
Model 1 -> t+1
Model 2 -> t+2
Model 3 -> t+3
...

This avoids feeding predictions back into the model.

But it increases the number of models and maintenance complexity.

44. Recursive vs Direct Forecasting

StrategyAdvantagesDisadvantages
RecursiveOne modelErrors can accumulate
DirectHorizon-specific modelsMore models to maintain
Multi-outputJoint predictionMore complex implementation
HybridFlexibleMore engineering complexity

The best approach depends on the forecast horizon and business requirements.

45. Multiple Seasonalities

Real-world series often have multiple seasonal patterns.

For example, hourly electricity demand may contain:

Mathematical Formulation
Daily:
24 hours

Weekly:
168 hours

Annual:
~8760 hours

We can create multiple lag features:

🐍 Python
df["lag_24"] = df["demand"].shift(24)

df["lag_168"] = df["demand"].shift(168)

We can also create multiple rolling windows.

46. Multiple Seasonal Features

For hourly data:

🐍 Python
seasonal_lags = [
 1,
 24,
 48,
 168
]

for lag in seasonal_lags:
 df[f"lag_{lag}"] = (
 df["demand"].shift(lag)
 )

The model can then learn interactions between:

  • Short-term behavior
  • Daily seasonality
  • Weekly seasonality

47. Time-Series Feature Design

A useful feature hierarchy is:

Architecture & Data Flow
Target History
 |
 +--> Lag 1
 +--> Lag 2
 +--> Lag 7
 +--> Lag 14
 +--> Lag 28

Recent Statistics
 |
 +--> Rolling Mean
 +--> Rolling Std
 +--> Rolling Min
 +--> Rolling Max

Calendar
 |
 +--> Hour
 +--> Day
 +--> Week
 +--> Month
 +--> Weekend

Business Context
 |
 +--> Price
 +--> Promotion
 +--> Holiday
 +--> Inventory
 +--> Weather

Not every project needs every feature.

Feature engineering should be guided by the problem.

48. Rolling Min and Max

🐍 Python
df["rolling_min_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .min()
)

df["rolling_max_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .max()
)

These can help describe the recent range of behavior.

49. Rolling Median

🐍 Python
df["rolling_median_7"] = (
 df["sales"]
 .shift(1)
 .rolling(7)
 .median()
)

Median-based features can sometimes be more robust to outliers than mean-based features.

50. Change Features

A model can also use recent changes.

One-step change:

Δyt=yt−yt−1\Delta y_t = y_t-y_{t-1}

Feature:

🐍 Python
df["change_1"] = (
 df["sales"].shift(1)
 -
 df["sales"].shift(2)
)

Seven-day change:

🐍 Python
df["change_7"] = (
 df["sales"].shift(1)
 -
 df["sales"].shift(8)
)

These can provide information about momentum and recent direction.

51. Percentage Change Features

🐍 Python
df["pct_change_1"] = (
 df["sales"]
 .shift(1)
 .pct_change()
)

Use percentage changes carefully when values can approach zero.

For some business problems, absolute changes are more meaningful.

52. Time-Based Cross-Validation

A single train/validation split may be insufficient.

Suppose we have:

Architecture & Data Flow
Fold 1:
Train -> Jan–Jun
Validation -> Jul

Fold 2:
Train -> Jan–Jul
Validation -> Aug

Fold 3:
Train -> Jan–Aug
Validation -> Sep

This is walk-forward or expanding-window validation.

It better reflects how forecasting models are used in practice.

Detailed time-series cross-validation will be covered in Notebook 5.

53. Why Standard K-Fold Is Wrong

Ordinary K-Fold may create:

Mathematical Formulation
Fold:
Training = future + past
Validation = middle

This violates the forecasting direction.

Time-series validation must preserve chronological order.

Use:

🐍 Python
from sklearn.model_selection import TimeSeriesSplit

Example:

🐍 Python
tscv = TimeSeriesSplit(
 n_splits=5
)

for train_idx, validation_idx in tscv.split(X):
 X_train_fold = X.iloc[train_idx]
 X_validation_fold = X.iloc[validation_idx]

 y_train_fold = y.iloc[train_idx]
 y_validation_fold = y.iloc[validation_idx]

54. Feature Engineering and Validation Leakage

There are two separate leakage risks.

Risk 1: Target Leakage#

Using future target values to construct features.

Risk 2: Transformation Leakage#

Calculating transformations using information from the future validation/test period.

For example, some preprocessing steps should be fitted only on training data.

A good principle is:

Any operation that learns parameters from data should be fitted using training information only.

55. Scaling Tree Models

Tree-based models generally do not require feature scaling.

For example:

🐍 Python
StandardScaler()

is usually unnecessary for:

  • Random Forest
  • XGBoost
  • LightGBM

This is different from many neural-network and distance-based algorithms.

However, scaling may still be relevant if you combine tree models with other algorithms.

56. Handling Missing Features

Lag and rolling features naturally create missing values at the beginning.

Example:

›lag_28

requires 28 previous observations.

A simple approach is:

🐍 Python
model_df = model_df.dropna()

This is often appropriate when the lost initial observations are small relative to the dataset.

For other missing values, use a strategy based on the meaning of the data.

57. Hyperparameter Tuning

Tree models contain many hyperparameters.

Examples:

Random Forest#

text
n_estimators
max_depth
min_samples_leaf
max_features

XGBoost#

text
n_estimators
learning_rate
max_depth
subsample
colsample_bytree

LightGBM#

text
n_estimators
learning_rate
num_leaves
max_depth
min_child_samples
subsample

Tuning should be performed using time-aware validation.

Do not randomly shuffle observations just to use ordinary cross-validation.

58. Early Stopping

Gradient boosting models can overfit when too many trees are added.

A validation set can help determine when to stop.

Conceptually:

text
Training error
 ↓
continues improving

Validation error
 ↓
improves
 ↓
reaches minimum
 ↓
starts worsening

The ideal point is near the validation minimum.

The exact early-stopping API differs between XGBoost and LightGBM versions, so check the installed library version before implementing it in production.

59. Model Comparison

A useful comparison table is:

ModelStrengthWeakness
NaiveExtremely simpleLimited
ARIMA/SARIMAInterpretable statistical structureMore assumptions
Random ForestRobust nonlinear baselineWeak extrapolation
XGBoostStrong nonlinear learningRequires tuning
LightGBMFast and scalableRequires tuning

The best model depends on:

  • Dataset size
  • Forecast horizon
  • Seasonality
  • External features
  • Nonlinearity
  • Computational budget
  • Interpretability requirements

60. Statistical Models vs Tree-Based ML

Statistical Approach#

Architecture & Data Flow
Temporal structure
 |
 v
Explicit statistical model
 |
 v
Forecast

Tree-Based ML#

Architecture & Data Flow
Temporal structure
 |
 v
Feature engineering
 |
 v
Tree-based model
 |
 v
Forecast

The ML approach is more flexible when many external variables and nonlinear interactions are available.

The statistical approach can be more elegant and interpretable when the temporal structure is well described by classical assumptions.

61. When Tree-Based ML Is Especially Useful

Tree-based ML can be attractive when:

  • Many external features exist.
  • Relationships are nonlinear.
  • There are many interactions.
  • Multiple seasonalities need to be represented.
  • You have rich calendar information.
  • The dataset is naturally tabular.
  • You want to reuse established supervised-learning tooling.

62. When Tree-Based ML May Struggle

Tree models may struggle with:

  • Long-term extrapolation.
  • Very long forecasting horizons.
  • Extremely high-dimensional sequential patterns.
  • Complex dependencies not represented in features.
  • Poorly designed lag features.

A tree model cannot learn temporal information that you never expose to it.

This is the central limitation of feature-based forecasting.

63. Forecast Visualization

After generating predictions:

🐍 Python
plt.figure(figsize=(14, 5))

plt.plot(
 y_test.index,
 y_test,
 label="Actual"
)

plt.plot(
 y_test.index,
 xgb_test_predictions,
 label="XGBoost"
)

plt.title(
 "Time Series ML Forecast"
)

plt.xlabel("Date")
plt.ylabel("Target")

plt.legend()

plt.show()

Always visualize predictions.

A metric alone can hide:

  • Systematic bias
  • Missed peaks
  • Missed seasonality
  • Delayed response
  • Forecast instability

64. Error Analysis

Create an error series:

🐍 Python
errors = (
 y_test
 - xgb_test_predictions
)

Plot:

🐍 Python
errors.plot(
 figsize=(14, 5)
)

plt.axhline(
 0,
 linestyle="--"
)

plt.title("Forecast Errors")
plt.show()

Ask:

  • Are errors centered around zero?
  • Are errors larger on weekends?
  • Are errors larger during peaks?
  • Does error variance change over time?

65. Error by Day of Week

Combine predictions with timestamps:

🐍 Python
error_df = pd.DataFrame({
 "actual": y_test,
 "prediction": xgb_test_predictions
})

error_df["error"] = (
 error_df["actual"]
 -
 error_df["prediction"]
)

error_df["day_of_week"] = (
 error_df.index.dayofweek
)

error_df.groupby(
 "day_of_week"
)["error"].mean()

This can reveal systematic calendar bias.

66. Error by Forecast Horizon

For multi-step forecasting, evaluate errors separately by horizon.

For example:

text
horizon 1
horizon 2
horizon 3
...
horizon 30

You may discover:

text
Short horizon:
Excellent

Long horizon:
Poor

This is one reason forecast-horizon design matters.

67. End-to-End Feature Engineering Pipeline

A practical workflow:

Architecture & Data Flow
Raw Time Series
 |
 v
Clean Timestamps
 |
 v
Sort Chronologically
 |
 v
Understand Frequency
 |
 v
Create Lag Features
 |
 v
Create Rolling Features
 |
 v
Create Calendar Features
 |
 v
Create External Features
 |
 v
Remove Invalid Rows
 |
 v
Chronological Split
 |
 v
Baseline
 |
 v
Random Forest
 |
 v
XGBoost
 |
 v
LightGBM
 |
 v
Time-Aware Validation
 |
 v
Error Analysis
 |
 v
Final Forecast

68. Reusable Training Function

🐍 Python
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import RandomForestRegressor


def evaluate_model(
 model,
 X_train,
 y_train,
 X_validation,
 y_validation
):
 model.fit(
 X_train,
 y_train
 )

 predictions = model.predict(
 X_validation
 )

 mae = mean_absolute_error(
 y_validation,
 predictions
 )

 return model, predictions, mae

Use:

🐍 Python
rf = RandomForestRegressor(
 n_estimators=300,
 min_samples_leaf=2,
 random_state=42,
 n_jobs=-1
)

rf, predictions, mae = evaluate_model(
 rf,
 X_train,
 y_train,
 X_validation,
 y_validation
)

print("Validation MAE:", mae)

69. Production-Oriented Feature Pipeline

A production forecasting system should conceptually separate:

Architecture & Data Flow
Raw Data
 |
 v
Validation
 |
 v
Feature Generation
 |
 v
Model
 |
 v
Prediction
 |
 v
Monitoring

The same feature-generation logic used during training must be reproducible during inference.

This is especially important for:

  • Lag features
  • Rolling statistics
  • Calendar features
  • External variables

A mismatch between training and inference features can cause serious production failures.

70. Common Beginner Mistakes

Mistake 1: Random Train/Test Split#

Use chronological splitting.

Mistake 2: Rolling Features Without Shift#

This can include the target in its own feature.

Mistake 3: Using Future External Data#

Only use information genuinely available at prediction time.

Mistake 4: Too Few Lags#

The model may not see important seasonal structure.

Mistake 5: Too Many Features#

More features do not automatically mean better forecasting.

Mistake 6: Ignoring Baselines#

Always compare against naive and seasonal-naive forecasts.

Mistake 7: Evaluating Only Training Performance#

Training accuracy is not forecasting performance.

Mistake 8: Using Standard K-Fold#

Use time-aware validation.

Mistake 9: Assuming Feature Importance Is Causality#

Predictive importance does not establish causal relationships.

71. Advanced Consideration: Target Leakage Through Aggregation

Suppose we are predicting daily sales.

If we calculate:

🐍 Python
monthly_total

using the full month, then for a prediction made halfway through the month we may accidentally include future days.

That is leakage.

The feature must represent only information available at the prediction timestamp.

This principle applies to:

  • Rolling statistics
  • Group aggregates
  • Monthly totals
  • Customer aggregates
  • External datasets
  • Normalization
  • Encodings

Always ask:

Could this value have been computed at prediction time?

72. Advanced Consideration: Forecast Availability

There is an important distinction between:

›Known future value

and:

›Unknown future value

For example:

Known#

text
Day of week
Holiday calendar
Month
Scheduled promotion

Forecasted / uncertain#

text
Future temperature
Future demand
Future competitor price

If an external variable is unknown at forecast time, you need either:

  • Its own forecast.
  • A scenario.
  • A known planned value.
  • A model that does not depend on it.

73. Advanced Consideration: Forecasting vs Interpolation

Forecasting:

›Past -> Future

Interpolation:

›Past -> Missing point -> Future

These are different problems.

Forecasting must not use future observations that would be unavailable at prediction time.

Interpolation can legitimately use both sides of a missing point because the task is reconstructing a historical observation.

Do not confuse the two.

74. Feature Engineering Strategy

A good feature-engineering process is iterative.

Start with:

text
lag_1
lag_7
rolling_mean_7
day_of_week

Then evaluate.

Add:

text
lag_14
lag_28
rolling_mean_28
rolling_std_7

Evaluate again.

Then add business features.

This makes it easier to understand which information actually improves forecasting.

75. Feature Ablation

Feature ablation means removing a feature group and measuring the performance change.

For example:

text
Model A:
Lags only

Model B:
Lags + rolling statistics

Model C:
Lags + rolling + calendar

Model D:
Lags + rolling + calendar + external

Compare:

text
MAE
RMSE
Forecast stability

This gives a much better understanding of feature value than simply inspecting feature importance.

76. Model Development Strategy

A practical sequence:

Architecture & Data Flow
1. Naive baseline
 |
2. Seasonal naive
 |
3. Simple lag-based model
 |
4. Random Forest
 |
5. XGBoost
 |
6. LightGBM
 |
7. Feature refinement
 |
8. Time-aware validation
 |
9. Final model

Do not jump immediately to the most complex algorithm.

77. Exercises

Exercise 1: Create Lag Features#

Using the sales dataset:

Create:

text
lag_1
lag_7
lag_14
lag_28

Explain what each represents.

Exercise 2: Rolling Features#

Create:

text
rolling_mean_7
rolling_mean_30
rolling_std_7
rolling_std_30

Make sure the current target is excluded from each feature.

Explain why .shift(1) matters.

Exercise 3: Calendar Features#

Create:

text
day_of_week
month
quarter
is_weekend
dow_sin
dow_cos

Compare model performance with and without calendar features.

Exercise 4: Model Comparison#

Train:

text
Naive
Random Forest
XGBoost
LightGBM

Compare:

  • MAE
  • RMSE

Then visualize all forecasts.

Exercise 5: Feature Ablation#

Create four models:

text
A: Lag features only

B: Lag + rolling

C: Lag + rolling + calendar

D: Lag + rolling + calendar + external

Compare their validation performance.

Explain which feature group provides the largest improvement.

78. Mini Project: Retail Demand Forecasting

Build an end-to-end forecasting system.

Part A: Data Preparation#

  • Parse timestamps.
  • Sort chronologically.
  • Identify frequency.
  • Handle missing dates.
  • Inspect missing values.

Part B: Feature Engineering#

Create:

  • Lag features.
  • Seasonal lags.
  • Rolling means.
  • Rolling standard deviations.
  • Calendar features.
  • Cyclical features.

Part C: Baselines#

Implement:

  • Naive forecast.
  • Seasonal-naive forecast.

Part D: Machine Learning#

Train:

  • Random Forest.
  • XGBoost.
  • LightGBM.

Part E: Validation#

Use chronological validation.

Compare:

  • MAE
  • RMSE

Part F: Analysis#

Investigate:

  • Feature importance.
  • Forecast errors.
  • Errors by weekday.
  • Errors during high-demand periods.

Part G: Final Recommendation#

Explain:

  1. Which model performed best?
  2. Which features mattered most?
  3. Did ML beat the baselines?
  4. Where did the model fail?
  5. What additional data could improve forecasting?

79. Key Takeaways

The central transformation is:

Architecture & Data Flow
Time Series
 |
 v
Temporal Feature Engineering
 |
 v
Supervised Learning
 |
 v
Tree-Based Model
 |
 v
Forecast

Remember:

  • Tree models do not inherently understand time.
  • Lag features expose historical information.
  • Rolling features summarize recent behavior.
  • Calendar features expose known temporal patterns.
  • Seasonal lags capture repeated behavior.
  • External variables can substantially improve forecasts.
  • .shift(1) is critical when constructing many one-step-ahead rolling features.
  • Forecasting must preserve temporal ordering.
  • Random train/test splitting can cause leakage.
  • Naive baselines should always be included.
  • Random Forest, XGBoost, and LightGBM are powerful tabular learners, but their success depends heavily on feature engineering.
  • Feature importance indicates predictive usefulness, not causality.
  • Multi-step forecasting requires a deliberate forecasting strategy.
  • Time-aware validation is essential.

80. Comparison With Notebook 2

At this point we have two fundamentally different approaches.

Statistical Forecasting#

Architecture & Data Flow
Series
 |
 v
Stationarity
 |
 v
ARIMA / SARIMA
 |
 v
Forecast

Machine Learning Forecasting#

Architecture & Data Flow
Series
 |
 v
Feature Engineering
 |
 v
Random Forest / XGBoost / LightGBM
 |
 v
Forecast

Neither approach is universally superior.

The correct question is:

Which approach provides the best reliable forecast for this specific problem?

That should be answered through proper validation.

81. Preparation for Notebook 4

The next notebook moves beyond manually engineered temporal features.

We will introduce deep-learning approaches for sequential data:

Architecture & Data Flow
Time Series
 |
 v
Sequence Windows
 |
 v
3D Tensor
(Samples, Timesteps, Features)
 |
 v
RNN
 |
 v
LSTM
 |
 v
GRU
 |
 v
Deep Sequence Forecasting

The key question will be:

Can a neural network learn temporal representations directly from sequences instead of relying primarily on manually engineered lag features?

Notebook 4 will cover:

  • Why standard feed-forward networks are limited for sequences.
  • RNN architecture.
  • Vanishing and exploding gradients.
  • LSTM gates and memory.
  • GRU architecture.
  • Sequence/window creation.
  • Scaling and inverse transformation.
  • 3D tensors.
  • TensorFlow/Keras implementation.
  • Training and validation.
  • Forecast generation.
  • Common deep-learning time-series mistakes.
Knowledge Checkpoint

ML for Time Series & Feature Engineering Checkpoint

Q1.Why can standard random cross-validation (KFold) NOT be applied directly to time series forecasting?
ARandom shuffling leaks future information into past training splits (temporal data leakage) and destroys temporal autocorrelation.
BBecause time series data is always 1-dimensional.
CBecause tree models cannot accept dates.
DBecause KFold only works on image data.
Q2.How are cyclical time features (such as hour of day 0-23 or day of year 1-365) correctly encoded for regression and tree models?
ASine and Cosine trigonometric transformations: $\sin(2\pi t / T)$ and $\cos(2\pi t / T)$
BOrdinal label encoding 1 to 24
COne-Hot encoding with 365 binary columns
DStandard MinMax scaling
Q3.What is the difference between a Rolling Window feature and an Expanding Window feature?
ARolling windows compute statistics over a fixed historical length (e.g. past 7 days), while expanding windows compute statistics over all available data from the start of the series to $t-1$.
BRolling windows are for classification; expanding windows are for clustering.
CRolling windows double in size every step.
DExpanding windows discard old data.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
Classical Statistical Time Series Models: ARIMA & SARIMA
Next Lesson
Deep Learning for Time Series: RNNs, LSTMs & GRUs