Notebook 1: Statistical Learning Foundations

Module: Machine Learning Level: Beginner Prerequisites: Notebook 0 (Introduction to Machine Learning), basic algebra Estimated Time: 3–4 hours

---

Table of Contents#

1. Why This Notebook Matters
2. Generalization: The Real Goal of ML
3. Underfitting and Overfitting
4. The Bias-Variance Tradeoff
5. Train / Validation / Test Splits
6. Cross-Validation
7. Hands-On: Visualizing Overfitting and the Bias-Variance Tradeoff
8. Learning Theory Basics
9. Case Studies
10. Common Misconceptions and Pitfalls
11. Glossary
12. Exercises
13. Quiz
14. Summary
15. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Explain why minimizing training loss is not the same as building a good model
• Diagnose underfitting and overfitting from training/validation curves
• Decompose prediction error into bias, variance, and irreducible error, and explain the tradeoff between them
• Correctly split data into training, validation, and test sets, and explain why each is needed
• Implement and interpret k-fold cross-validation
• State, at a basic level, why learning theory bounds generalization error in terms of model complexity and sample size
• Recognize data leakage and other common evaluation mistakes before they corrupt your results

---

1. Why This Notebook Matters#

In Notebook 0, Section 4.4, we defined training as Empirical Risk Minimization (ERM):

$\hat{\theta} = \arg\min_{\theta} \frac{1}{n}\sum_{i=1}^{n} L\big(y^{(i)}, h_\theta(x^{(i)})\big)$

This equation only tells the model to perform well on the n examples it has already seen. But a model that has only ever seen 1,000 house prices is not useful because it can perfectly recall those 1,000 prices — it is useful because it can predict the price of house number 1,001, which it has never seen.

This notebook is about closing the gap between two very different quantities:

• Training error: how well the model does on the data it learned from
• Generalization error: how well the model does on new, unseen data

Everything else in this course — every algorithm, every evaluation metric, every tuning strategy — exists in service of minimizing generalization error, not training error. Get comfortable with that distinction now; it is the single most important idea in applied machine learning.

---

2. Generalization: The Real Goal of ML#

2.1 The Data Generating Process#

Assume that both the training data and any future data are drawn independently from the same fixed but unknown probability distribution:

$(x^{(i)}, y^{(i)}) \overset{\text{i.i.d.}}{\sim} P(X, Y)$

The true risk (also called the population risk or expected risk) of a model $h_\theta$ is its average loss over the entire distribution, not just the training sample:

$R(\theta) = \mathbb{E}_{(x,y)\sim P}\big[L(y, h_\theta(x))\big]$

This is the quantity we actually care about, but we can never compute it directly — we do not have access to the full distribution $P$, only a finite sample from it. So we approximate $R(\theta)$ with the empirical risk from Notebook 0:

$\hat{R}(\theta) = \frac{1}{n}\sum_{i=1}^{n} L\big(y^{(i)}, h_\theta(x^{(i)})\big)$

The generalization gap is the difference between these two quantities:

$\text{Generalization Gap} = R(\theta) - \hat{R}(\theta)$

A small training loss with a large generalization gap means the model has essentially memorized the training set without learning the underlying pattern — this is overfitting, covered next.

2.2 Why This Gap Exists#

The gap exists because $\hat{R}(\theta)$ is computed on a finite sample. With enough parameters, a model can drive $\hat{R}(\theta)$ arbitrarily close to zero by fitting every quirk and noise point in the training sample — quirks that will not reappear in new data drawn from $P$. The rest of this notebook is about controlling that gap.

---

3. Underfitting and Overfitting#

3.1 Definitions#

• Underfitting: the model is too simple to capture the underlying pattern in the data. Both training error and generalization error are high.
• Overfitting: the model is too complex relative to the amount of training data. Training error is very low, but generalization error is high — the model has fit noise specific to the training sample rather than the true underlying signal.
• Good fit: the model captures the true underlying pattern without fitting noise. Training error and generalization error are both reasonably low and close to each other.

3.2 A Concrete Illustration#

Suppose the true relationship is a slightly curved line, but we only observe it with noise. Consider three models:

The degree-15 model achieves almost perfect training accuracy, which sounds impressive, but it performs worse than the simple degree-3 model on new data. This is the central paradox that this notebook resolves: lower training error does not imply a better model.

3.3 The Textbook Curve#

Plotting error against model complexity produces one of the most important diagrams in all of machine learning:

Architecture & Data Flow
Copy
   Error
     |                                    ____ Test/Validation Error
     |                                  /
     |                               /
     |                            /
     |                        /
     |___________ Training Error ______________
     |      \_______________________________
     |
     +----------------------------------------------> Model Complexity
          Underfitting    Sweet Spot    Overfitting

Training error decreases monotonically as complexity increases (a more flexible model can always fit the training data at least as well). Validation/test error decreases at first, reaches a minimum, then increases again. The goal is to find the complexity level at the minimum of the validation curve, not the minimum of the training curve.

---

4. The Bias-Variance Tradeoff#

4.1 Formal Decomposition#

Consider a regression setting where the true relationship is $y = f(x) + \epsilon$, with $\epsilon$ representing irreducible noise with mean zero and variance $\sigma^2$. For a model $\hat f$ trained on a random sample, the expected squared error at a point $x$ decomposes as:

$\mathbb{E}\big[(y - \hat f(x))^2\big] = \underbrace{\big(\text{Bias}[\hat f(x)]\big)^2}_{\text{bias}^2} + \underbrace{\text{Var}[\hat f(x)]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{irreducible error}}$

where:

$\text{Bias}[\hat f(x)] = \mathbb{E}[\hat f(x)] - f(x)$ $\text{Var}[\hat f(x)] = \mathbb{E}\Big[\big(\hat f(x) - \mathbb{E}[\hat f(x)]\big)^2\Big]$

The expectation here is taken over many hypothetical training sets drawn from the same distribution — imagine retraining the model on 1,000 different random samples of the same size and looking at how the predictions vary.

4.2 Interpreting Each Term#

• Bias: error from overly simplistic assumptions in the model. A high-bias model (e.g., fitting a straight line to curved data) systematically misses the true pattern regardless of how much data you give it. High bias is associated with underfitting.
• Variance: error from the model's sensitivity to the specific training sample. A high-variance model (e.g., a very deep decision tree) produces wildly different predictions if trained on a slightly different sample of the same size. High variance is associated with overfitting.
• Irreducible error ($\sigma^2$): noise inherent in the problem itself. No model, however good, can eliminate this — it represents the theoretical floor on achievable error.

4.3 The Tradeoff#

Model complexity controls a dial between bias and variance:

You cannot reduce both simultaneously just by changing complexity — reducing bias by adding complexity tends to increase variance, and vice versa. The best achievable model minimizes the sum of bias-squared and variance, not either term in isolation. Techniques covered later in the course (regularization in Notebook 3/5, ensembling in Notebook 6) are specifically designed to reduce variance without a proportional increase in bias, effectively getting a better tradeoff than complexity control alone.

4.4 A Numeric Intuition#

Imagine four archers shooting at a target, where the bullseye is the true value $f(x)$:

• Low bias, low variance: arrows cluster tightly around the bullseye. This is the ideal model.
• Low bias, high variance: arrows are centered around the bullseye on average, but scattered widely. This is an overfit model — correct "on average" across many retrainings, but unreliable for any single trained instance.
• High bias, low variance: arrows cluster tightly, but off to one side, away from the bullseye. This is an underfit model — consistent, but consistently wrong.
• High bias, high variance: arrows are scattered and off-target. This is the worst case — a poorly chosen model with insufficient data.

---

5. Train / Validation / Test Splits#

5.1 Why One Dataset Is Not Enough#

If you train a model and then evaluate it on the same data it was trained on, you are measuring $\hat{R}(\theta)$, the empirical risk — which we already established can be made arbitrarily low by overfitting. To estimate the true risk $R(\theta)$, you need data the model has never seen during training.

5.2 The Three-Way Split#

A critical rule: the test set must be touched exactly once, at the very end, after all model and hyperparameter decisions have been finalized using only the training and validation sets. If you look at test performance multiple times and adjust your approach in response, you are implicitly fitting to the test set, and its performance estimate becomes optimistically biased — this is a subtle form of overfitting at the experimental level, not just the model level.

5.3 Why a Separate Validation Set (Not Just Test)#

You might ask: why not just use the test set for hyperparameter tuning? Because tuning is itself a form of fitting — trying many hyperparameter combinations and picking the one that performs best on a given dataset will optimistically bias your estimate of performance on that dataset. The validation set exists so this bias contaminates a "disposable" set, leaving the test set clean for a final, honest estimate.

5.4 Code Example#

Python
CopyAsk AI
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes

X, y = load_diabetes(return_X_y=True)

# First split: separate out the test set (held back until the very end)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42
)

# Second split: divide the remainder into train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.1765, random_state=42
    # 0.1765 * 0.85 ≈ 0.15, giving roughly 70/15/15 overall
)

print(f"Train: {len(X_train)} | Validation: {len(X_val)} | Test: {len(X_test)}")

Expected output:

Train: 309 | Validation: 66 | Test: 67

---

6. Cross-Validation#

6.1 The Problem With a Single Validation Split#

A single train/validation split gives you one estimate of performance, which depends on which specific examples happened to land in each split. With a small dataset, this estimate can be noisy and unreliable — a "lucky" or "unlucky" split can make a model look better or worse than it really is.

6.2 K-Fold Cross-Validation#

K-fold cross-validation addresses this by using every example for both training and validation, across multiple rounds:

1. Split the training data into $k$ equally sized folds (commonly $k=5$ or $k=10$).
2. For each fold $i \in \{1, \dots, k\}$:
   • Train the model on all folds except fold $i$
   • Evaluate it on fold $i$
3. Average the $k$ resulting scores to get the final cross-validation estimate.

code
Ask AICopy
Fold 1: [ VALID ][ train ][ train ][ train ][ train ]
Fold 2: [ train ][ VALID ][ train ][ train ][ train ]
Fold 3: [ train ][ train ][ VALID ][ train ][ train ]
Fold 4: [ train ][ train ][ train ][ VALID ][ train ]
Fold 5: [ train ][ train ][ train ][ train ][ VALID ]

Mathematically, the cross-validation estimate of risk is:

$\text{CV}(k) = \frac{1}{k}\sum_{i=1}^{k} \hat{R}_i(\theta_{-i})$

where $\theta_{-i}$ denotes parameters trained without fold $i$, and $\hat{R}_i$ is the empirical risk measured on fold $i$ alone.

6.3 Why Cross-Validation Helps#

• It uses all of the data for both training and validation at some point, which is especially valuable with small datasets.
• Averaging over $k$ folds produces a much more stable estimate of generalization performance than a single split, and also provides a variance estimate (how much the score fluctuates across folds), which tells you how much to trust the average.

6.4 Stratified K-Fold#

For classification problems, especially with imbalanced classes, stratified k-fold ensures each fold preserves the same class proportions as the full dataset. Without this, a fold could end up with very few (or zero) examples of a rare class, producing a misleading evaluation.

6.5 Code Example#

Python
CopyAsk AI
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
import numpy as np

X, y = load_diabetes(return_X_y=True)

model = Ridge(alpha=1.0)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=kfold, scoring="neg_mean_squared_error")
mse_scores = -scores  # sklearn returns negative MSE by convention

print(f"MSE per fold: {np.round(mse_scores, 1)}")
print(f"Mean MSE: {mse_scores.mean():.1f} (+/- {mse_scores.std():.1f})")

Expected output (approximate):

Mathematical Formulation
Explain FormulaCopy
MSE per fold: [2999.  3178.  2841.9 2864.  2933.5]
Mean MSE: 2963.3 (+/- 116.4)

6.6 Computational Cost#

Cross-validation multiplies training cost by $k$, since the model is retrained $k$ times. For very large datasets or expensive models, this cost can be significant, and a single held-out validation split (or a smaller $k$, such as 3) may be a practical compromise.

---

7. Hands-On: Visualizing Overfitting and the Bias-Variance Tradeoff#

This example fits polynomials of increasing degree to noisy data and plots training versus validation error, directly reproducing the curve from Section 3.3.

Python
CopyAsk AI
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

# Generate synthetic data from a true nonlinear function plus noise
rng = np.random.RandomState(42)
X = np.sort(rng.uniform(0, 1, 60)).reshape(-1, 1)
y_true = np.sin(2 * np.pi * X.ravel())
y = y_true + rng.normal(scale=0.2, size=X.shape[0])

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)

degrees = [1, 3, 5, 8, 12, 15]
print(f"{'Degree':>6} | {'Train MSE':>10} | {'Val MSE':>10}")
print("-" * 32)

for degree in degrees:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X_train, y_train)

    train_mse = mean_squared_error(y_train, model.predict(X_train))
    val_mse = mean_squared_error(y_val, model.predict(X_val))

    print(f"{degree:>6} | {train_mse:>10.4f} | {val_mse:>10.4f}")

Expected output (approximate, will vary slightly by environment):

code
Ask AICopy
Degree | Train MSE |    Val MSE
--------------------------------
     1 |     0.4312 |     0.5464
     3 |     0.0367 |     0.0516
     5 |     0.0339 |     0.0498
     8 |     0.0301 |     0.0678
    12 |     0.0198 |     0.1584
    15 |     0.0104 |     0.4931

Notice how training MSE decreases steadily as degree increases, while validation MSE bottoms out around degree 3-5 and then rises sharply — a direct, numeric demonstration of the underfitting/overfitting curve from Section 3.3, and of the bias-variance tradeoff from Section 4.

---

8. Learning Theory Basics#

This section gives an intuitive, non-rigorous introduction to why generalization is even mathematically possible. A full treatment (PAC learning, VC dimension) belongs in a dedicated statistical learning theory course, but the core intuition is valuable here.

8.1 The Central Question#

Given a finite training sample, why should we expect a model's performance on that sample to say anything at all about its performance on new data? Statistical learning theory answers this with concentration inequalities, which bound the probability that empirical risk deviates far from true risk, as a function of sample size $n$ and model complexity.

8.2 A Simplified Generalization Bound#

One classical result (informally stated) says that, with high probability, for every hypothesis $h$ in a hypothesis space $\mathcal{H}$:

$R(h) \leq \hat{R}(h) + O\left(\sqrt{\frac{\text{complexity}(\mathcal{H})}{n}}\right)$

Read this as: true risk is bounded by empirical risk, plus a penalty term that grows with model complexity and shrinks as sample size $n$ grows.

This single inequality formally captures everything discussed so far in this notebook:

• If $\text{complexity}(\mathcal{H})$ is very high (e.g., a very deep tree, a very high-degree polynomial) relative to $n$, the penalty term is large, and low training error offers little guarantee about true risk. This is overfitting, in formal terms.
• As $n$ grows, the penalty term shrinks toward zero for any fixed hypothesis space, meaning more data narrows the generalization gap. This is why "get more data" is one of the most reliable ways to combat overfitting.
• "Complexity" here can be formalized in different ways depending on the model family — the VC dimension is one classical measure, roughly capturing how many different labelings of points a hypothesis space can represent.

8.3 Why This Matters in Practice#

You do not need to compute VC dimensions to build good models day-to-day, but the intuition behind this bound explains several practices you will use throughout this course:

• Regularization (Notebooks 3, 5) works by effectively shrinking the usable hypothesis space, reducing the complexity penalty.
• Ensembling (Notebook 6) works by averaging multiple high-variance models, which reduces variance (Section 4) without increasing bias much.
• Collecting more data directly shrinks the generalization gap for a fixed model complexity, per the bound above.
• Simpler models tend to generalize better with small datasets, precisely because the complexity penalty dominates when $n$ is small.

---

9. Case Studies#

9.1 Case Study: Medical Diagnosis Model Overfitting to Hospital-Specific Noise#

A team trains a model to predict disease risk using data from a single hospital, achieving 98 percent training accuracy. When deployed at a different hospital, accuracy drops to 62 percent.

Diagnosis: The model likely overfit to hospital-specific artifacts — a particular lab machine's calibration, a particular scanning protocol, or even timestamp patterns correlated with which physician was on duty — rather than the true clinical signal. A proper train/validation/test split within the original hospital's data would not have caught this, because all three splits shared the same hospital-specific noise. This illustrates a specific and dangerous failure mode: your held-out data must represent the true deployment distribution, not just a random subset of the same convenient sample.

Lesson: Cross-validation and train/test splits protect against overfitting to specific examples, but not against overfitting to systematic biases shared across your entire dataset. Validating on genuinely external data (a different hospital, a different time period) is necessary to catch this.

9.2 Case Study: Kaggle Leaderboard Overfitting#

Public data science competitions often have a public leaderboard (scored on a subset of test data) and a private leaderboard (revealed only at the end, scored on the remaining test data). It is a well-documented phenomenon that competitors who repeatedly tune their models to maximize public leaderboard score often see a significant drop in rank on the private leaderboard.

Diagnosis: This is exactly the phenomenon described in Section 5.2 — repeated evaluation against the same held-out set, followed by adjustments based on that feedback, causes implicit overfitting to that set, even without the model ever training on it directly.

Lesson: This is precisely why the guidance in Section 5.2 says the test set must be touched exactly once. Real-world model development should rely on cross-validation or a validation set for all iteration, reserving a genuine test set for a final check.

9.3 Case Study: Bias-Variance in A/B Testing Personalization Models#

An e-commerce company compares two recommendation models: Model A (a simple popularity-based ranker) and Model B (a complex model with millions of parameters trained on limited per-user interaction data). In offline testing, Model B shows lower training error. In a live A/B test, Model A performs better on the smaller user segments (new users with little history), while Model B performs better on the largest, most active user segment.

Diagnosis: For new users with sparse interaction history, Model B's high variance causes it to fit noise from the few observed interactions, producing unreliable recommendations (high variance regime). Model A's simplicity, while biased, is far more stable given very little data per user. For highly active users with abundant interaction history, Model B's flexibility allows it to capture real personalized patterns that Model A's simplicity misses (bias dominates for Model A here).

Lesson: The right point on the bias-variance tradeoff is not a universal constant — it depends on how much relevant data is available for the specific prediction being made, which can vary within a single deployed system.

---

10. Common Misconceptions and Pitfalls#

---

11. Glossary#

---

12. Exercises#

Exercise 1 (Conceptual)#

A colleague reports: "My model gets 99.8 percent accuracy on the training set, so I'm confident it will perform well in production." What is wrong with this reasoning, and what would you ask to see before agreeing or disagreeing?

<details> <summary>Solution</summary>

Training accuracy alone says nothing about generalization (Section 2). A very high training accuracy combined with an unknown or unreported validation/test accuracy is actually a warning sign for overfitting, not a guarantee of quality. Before agreeing, ask to see performance on a held-out validation or test set that was not used during training or tuning, ideally using cross-validation for a more stable estimate, and check whether that held-out data genuinely represents the production distribution (see the medical diagnosis case study in Section 9.1).

</details>

---

Exercise 2 (Conceptual)#

Classify each scenario as primarily a bias problem or a primarily a variance problem, and justify your answer:

a) A linear regression model applied to data with a strong quadratic relationship, performing poorly on both training and test sets. b) A decision tree with no depth limit, achieving near-zero training error but poor test error.

<details> <summary>Solution</summary>

a) Bias problem. The model family (a straight line) cannot represent the true quadratic relationship no matter how much data is provided; both training and test error are high because the model is systematically wrong everywhere. This matches the "high bias, underfitting" pattern from Section 4.3.

b) Variance problem. The unlimited-depth tree can perfectly memorize training data (near-zero training error) but is highly sensitive to the specific training sample, producing a model that does not generalize. This matches the "low bias, high variance, overfitting" pattern from Section 4.3.

</details>

---

Exercise 3 (Coding)#

Using the load_diabetes dataset, compare 5-fold cross-validation MSE for Ridge(alpha=0.1) versus Ridge(alpha=100). Which one appears to overfit more, based on the fold-to-fold variability of the scores?

<details> <summary>Solution</summary>

Python
CopyAsk AI
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import Ridge
from sklearn.datasets import load_diabetes
import numpy as np

X, y = load_diabetes(return_X_y=True)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)

for alpha in [0.1, 100]:
    model = Ridge(alpha=alpha)
    scores = -cross_val_score(model, X, y, cv=kfold, scoring="neg_mean_squared_error")
    print(f"alpha={alpha}: mean MSE={scores.mean():.1f}, std={scores.std():.1f}")

A very small alpha (weak regularization) tends to behave more like ordinary linear regression, potentially fitting more noise (higher variance across folds), while a very large alpha (strong regularization) tends to shrink coefficients aggressively, which can underfit and increase bias. Comparing the mean MSE and the standard deviation across folds for each setting will reveal which regime this particular dataset favors, and this exact tradeoff, explored through the alpha hyperparameter, is covered in depth in Notebook 5 (Regularization Techniques).

</details>

---

Exercise 4 (Applied Thinking)#

You are building a fraud detection model and you have transaction data spanning the last three years. Describe a train/validation/test splitting strategy, and explain why a purely random split might be a mistake in this scenario.

<details> <summary>Solution</summary>

Because fraud patterns evolve over time (as mentioned in the fraud detection case study in Notebook 0), a purely random split could place transactions from the same time period, and even the same short-lived fraud campaign, into both the training and test sets. This would let the model "see" patterns from a fraud campaign during training and then get evaluated on near-identical examples from the same campaign, inflating the test score in a way that will not hold up on genuinely future, unseen fraud patterns. A better strategy is a time-based split: train on the earliest period, validate on a middle period, and test on the most recent period, so evaluation always mimics the real deployment scenario of predicting the future from the past.

</details>

---

13. Quiz#

Q1. A model achieves near-zero error on training data but performs poorly on new data. This is best described as: a) Underfitting b) Overfitting c) High bias d) Irreducible error

Q2. In the bias-variance decomposition, which term can never be reduced to zero, no matter how good the model is? a) Bias b) Variance c) Irreducible error d) Training error

Q3. What is the primary purpose of a validation set, as distinct from a test set? a) To train the model's parameters b) To tune hyperparameters and select between models before final evaluation c) To provide a completely unbiased, one-time final performance estimate d) To increase the size of the training set

Q4. In k-fold cross-validation with k=5, how many times is the model trained in total? a) 1 b) 5 c) 25 d) It depends on the dataset size

Q5. True or False: Touching the test set multiple times during model development, and adjusting your approach based on the results each time, does not bias your final performance estimate as long as you never train directly on the test data.

<details> <summary>Answers</summary>

1. b) Overfitting
2. c) Irreducible error
3. b) To tune hyperparameters and select between models before final evaluation
4. b) 5 (once per fold, each time training on the remaining k-1 folds)
5. False — repeated evaluation followed by adjustment leaks information from the test set into your modeling decisions, biasing the final estimate optimistically, as discussed in Section 5.2 and the Kaggle leaderboard case study in Section 9.2

</details>

---

14. Summary#

• The real goal of machine learning is minimizing generalization error (true risk), not training error (empirical risk); these are different quantities that can diverge sharply.
• Underfitting (high bias) and overfitting (high variance) are the two failure modes on either side of the ideal model complexity, visualized by the training-versus-validation error curve.
• The bias-variance decomposition formally splits expected prediction error into bias-squared, variance, and irreducible error, explaining why increasing model flexibility is not a free lunch.
• Train/validation/test splits exist specifically to produce honest estimates of generalization performance, with the test set reserved for a single, final check.
• K-fold cross-validation produces a more stable and data-efficient estimate of generalization performance than a single validation split, and stratified k-fold preserves class balance for classification problems.
• Learning theory formally explains why more data narrows the generalization gap, and why the "right" amount of model complexity depends on how much data is available.
• Data leakage, subtle or obvious, is one of the most common and dangerous mistakes in applied machine learning, and often arises from applying preprocessing steps before splitting the data.

---

15. Further Reading#

• Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements of Statistical Learning, Chapter 7 (Model Assessment and Selection)
• Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms — a rigorous but accessible treatment of PAC learning and VC dimension
• scikit-learn documentation on cross-validation: https://scikit-learn.org/stable/modules/cross_validation.html
• scikit-learn documentation on learning curves: https://scikit-learn.org/stable/modules/learning_curve.html

---

Next Up#

Notebook 2: Data Preprocessing and Feature Engineering — scaling and normalization, encoding categorical variables, handling missing data, feature selection, and feature engineering techniques. Correct preprocessing is also where data leakage most commonly creeps in, directly building on Section 10 of this notebook.

---

Course: Machine Learning | Notebook 1 of 13