Notebook 3: Linear Regression and Regularization

Module: Machine Learning Level: Beginner to Intermediate Prerequisites: Notebook 1 (Statistical Learning Foundations), Notebook 2 (Data Preprocessing and Feature Engineering) Estimated Time: 3–4 hours

---

Table of Contents#

1. Why Start With Linear Regression
2. Simple Linear Regression
3. The Ordinary Least Squares Solution
4. Multiple Linear Regression
5. The Assumptions Behind OLS
6. Diagnosing Assumption Violations
7. Why Regularize: Overfitting in Linear Models
8. Ridge Regression (L2 Regularization)
9. Lasso Regression (L1 Regularization)
10. Elastic Net
11. Hands-On: Comparing OLS, Ridge, and Lasso
12. Case Studies
13. Common Misconceptions and Pitfalls
14. Glossary
15. Exercises
16. Quiz
17. Summary
18. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Derive and interpret the Ordinary Least Squares solution for both simple and multiple linear regression
• State the five classical assumptions behind OLS and explain the consequence of violating each
• Diagnose assumption violations using residual plots and other diagnostic tools
• Explain why linear regression can overfit, particularly with many features or multicollinearity
• Derive and apply Ridge (L2) and Lasso (L1) regularization, and explain their different effects on coefficients
• Choose between Ridge, Lasso, and Elastic Net based on the structure of a given problem
• Fit, tune, and interpret regularized linear models in scikit-learn

---

1. Why Start With Linear Regression#

Linear regression is the oldest, simplest, and most interpretable model family in the classical ML toolkit, dating back to Gauss and Legendre in the early 1800s. It is included first in this course not because it is the most powerful algorithm, but because nearly every idea introduced here reappears, in generalized form, throughout the rest of the course: the loss function reappears in logistic regression (Notebook 4), the regularization concepts reappear in every model family that supports them, and the coefficient interpretation techniques form the baseline against which more complex, less interpretable models are compared.

Recall the hypothesis space for linear models from Notebook 0, Section 4.2:

$\mathcal{H} = \{h_\theta(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_d x_d \mid \theta \in \mathbb{R}^{d+1}\}$

This notebook is entirely about how to find the best $\theta$ within this hypothesis space, and how to control its complexity when the data does not support a fully flexible fit.

---

2. Simple Linear Regression#

2.1 The Model#

With a single feature $x$, the model is a straight line:

$\hat{y} = \theta_0 + \theta_1 x$

where $\theta_0$ is the intercept (the predicted value when $x = 0$) and $\theta_1$ is the slope (the change in predicted $y$ for a one-unit increase in $x$).

2.2 The Loss Function#

Following the Empirical Risk Minimization framework from Notebook 0, Section 4.4, linear regression uses the squared error loss:

$L(y, \hat{y}) = (y - \hat{y})^2$

The empirical risk to minimize, commonly called the cost function $J(\theta)$ in this context, is:

$J(\theta_0, \theta_1) = \frac{1}{n}\sum_{i=1}^{n} \big(y^{(i)} - (\theta_0 + \theta_1 x^{(i)})\big)^2$

Some textbooks use $\frac{1}{2n}$ instead of $\frac{1}{n}$ for cleaner derivative expressions; the choice of constant does not affect which $\theta$ minimizes the function.

---

3. The Ordinary Least Squares Solution#

3.1 Why "Least Squares"#

The method that minimizes $J(\theta)$ is called Ordinary Least Squares (OLS) because it finds the parameters that minimize the sum of squared vertical distances (residuals) between the observed points and the fitted line.

3.2 The Closed-Form Solution (Simple Case)#

For simple linear regression, calculus gives a closed-form solution by taking partial derivatives of $J$ with respect to $\theta_0$ and $\theta_1$, setting them to zero, and solving:

$\theta_1 = \frac{\sum_{i=1}^{n}(x^{(i)} - \bar{x})(y^{(i)} - \bar{y})}{\sum_{i=1}^{n}(x^{(i)} - \bar{x})^2}$

$\theta_0 = \bar{y} - \theta_1 \bar{x}$

where $\bar{x}$ and $\bar{y}$ are the sample means of $x$ and $y$.

3.3 The Matrix Form (General Case)#

For multiple features, it is cleaner to work in matrix notation. Let $X$ be the $n \times (d+1)$ design matrix (with a column of ones prepended for the intercept), and $y$ be the $n \times 1$ vector of targets. The cost function becomes:

$J(\theta) = \frac{1}{n}\|y - X\theta\|^2$

Setting the gradient with respect to $\theta$ to zero and solving yields the Normal Equation:

$\hat{\theta} = (X^T X)^{-1} X^T y$

This closed-form solution exists whenever $X^T X$ is invertible. Section 7 discusses precisely what happens, and why regularization becomes necessary, when this matrix is close to non-invertible.

3.4 Gradient Descent as an Alternative#

For very large datasets, computing $(X^T X)^{-1}$ directly can be computationally expensive (it scales roughly as $O(d^3)$ in the number of features). An iterative alternative, gradient descent, updates $\theta$ step by step in the direction that most decreases $J(\theta)$:

$\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta)$

where $\alpha$ is the learning rate, a hyperparameter controlling step size. Gradient descent is covered in more operational depth in Notebook 4 (Logistic Regression), where, unlike OLS, no closed-form solution exists at all.

3.5 Code Example#

Python
CopyAsk AI
import numpy as np
from sklearn.linear_model import LinearRegression

# Simple linear regression: predicting exam score from hours studied
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
scores = np.array([52, 58, 63, 70, 74, 80, 85, 91])

model = LinearRegression()
model.fit(hours, scores)

print(f"Intercept (theta_0): {model.intercept_:.2f}")
print(f"Slope (theta_1): {model.coef_[0]:.2f}")
print(f"Predicted score for 9 hours: {model.predict([[9]])[0]:.2f}")

Expected output:

code
Ask AICopy
Intercept (theta_0): 46.68
Slope (theta_1): 5.60
Predicted score for 9 hours: 97.09

Interpretation: Each additional hour studied is associated with an estimated 5.60-point increase in exam score, holding nothing else constant since this is a single-feature model.

---

4. Multiple Linear Regression#

4.1 The Model#

With $d$ features, the model generalizes directly:

$\hat{y} = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_d x_d$

4.2 Interpreting Coefficients#

In multiple regression, each coefficient $\theta_j$ represents the expected change in $y$ for a one-unit increase in $x_j$, holding all other features constant. This "holding constant" clause is essential and is a common source of misinterpretation, discussed further in Section 13.

4.3 Code Example#

Python
CopyAsk AI
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

data = pd.DataFrame({
    "sqft": [1400, 1600, 1700, 1875, 1100, 2350, 2450, 1425],
    "bedrooms": [3, 3, 3, 4, 2, 4, 4, 3],
    "age_years": [15, 8, 20, 3, 30, 5, 2, 12],
    "price": [245000, 312000, 279000, 358000, 195000, 412000, 435000, 268000]
})

X = data[["sqft", "bedrooms", "age_years"]]
y = data["price"]

model = LinearRegression()
model.fit(X, y)

for feature, coef in zip(X.columns, model.coef_):
    print(f"{feature}: {coef:.2f}")
print(f"Intercept: {model.intercept_:.2f}")

Expected output (illustrative, will vary with this small sample):

code
Ask AICopy
sqft: 148.32
bedrooms: 8210.55
age_years: -1875.40
Intercept: 12500.10

Interpretation: Holding bedrooms and age constant, each additional square foot is associated with roughly 148 dollars in additional price. Holding sqft and bedrooms constant, each additional year of age is associated with roughly 1,875 dollars less in price.

---

5. The Assumptions Behind OLS#

OLS produces statistically well-behaved estimates (specifically, the Best Linear Unbiased Estimator, per the Gauss-Markov theorem) only when certain assumptions hold. Understanding these assumptions is essential for knowing when linear regression is appropriate and when its coefficient interpretations and confidence intervals can be trusted.

---

6. Diagnosing Assumption Violations#

6.1 Residual Plots#

A residual is the difference between the observed and predicted value: $e^{(i)} = y^{(i)} - \hat{y}^{(i)}$. Plotting residuals against predicted values (or against each feature) is the single most useful diagnostic tool:

• Random scatter around zero, constant spread: consistent with linearity and homoscedasticity holding.
• A curved pattern: suggests a nonlinear relationship is present (linearity violated) — consider polynomial features or a nonlinear model (Notebooks 6-8).
• A funnel/cone shape (spread increasing or decreasing): suggests heteroscedasticity — consider a log transform of the target, or weighted least squares.

Python
CopyAsk AI
import matplotlib.pyplot as plt

residuals = y_test - model.predict(X_test)
plt.scatter(model.predict(X_test), residuals)
plt.axhline(y=0, color="black", linestyle="--")
plt.xlabel("Predicted values")
plt.ylabel("Residuals")
plt.title("Residual Plot")

6.2 Detecting Multicollinearity: Variance Inflation Factor#

The Variance Inflation Factor (VIF) quantifies how much a coefficient's variance is inflated due to correlation with other features:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

where $R_j^2$ is the R-squared from regressing feature $j$ on all other features. A common rule of thumb treats VIF above 5 or 10 as indicating problematic multicollinearity.

Python
CopyAsk AI
from statsmodels.stats.outliers_influence import variance_inflation_factor
import pandas as pd

vif_data = pd.DataFrame()
vif_data["feature"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)

6.3 Quantile-Quantile (Q-Q) Plots#

A Q-Q plot compares the distribution of residuals against a theoretical normal distribution. Points falling closely along a straight diagonal line support the normality assumption; systematic deviation, especially in the tails, suggests non-normal residuals.

---

7. Why Regularize: Overfitting in Linear Models#

7.1 The Multicollinearity and High-Dimensionality Problem#

Recall the Normal Equation from Section 3.3: $\hat{\theta} = (X^T X)^{-1} X^T y$. This solution requires $X^T X$ to be invertible. Two common real-world situations break this assumption or make it numerically fragile:

• High multicollinearity: when features are highly correlated, $X^T X$ becomes close to singular (non-invertible), causing the OLS solution to become extremely sensitive to small changes in the data — a textbook case of high variance, per Notebook 1, Section 4.
• More features than observations ($d > n$): $X^T X$ is mathematically guaranteed to be non-invertible, and the Normal Equation has no unique solution at all.

7.2 Regularization as a Bias-Variance Tradeoff Tool#

Regularization directly addresses this by adding a penalty term to the cost function that discourages large coefficient values:

$J_{\text{regularized}}(\theta) = \underbrace{\frac{1}{n}\|y - X\theta\|^2}_{\text{original loss}} + \underbrace{\lambda \cdot \text{penalty}(\theta)}_{\text{regularization term}}$

where $\lambda \geq 0$ (sometimes written $\alpha$ in scikit-learn) controls the strength of the penalty. This is a direct, practical application of the bias-variance tradeoff from Notebook 1, Section 4: increasing $\lambda$ shrinks coefficients toward zero, which increases bias slightly but can substantially reduce variance, often improving generalization overall, per the learning theory bound discussed in Notebook 1, Section 8.2 (regularization effectively shrinks the usable hypothesis space).

---

8. Ridge Regression (L2 Regularization)#

8.1 The Formula#

Ridge regression adds a penalty proportional to the sum of squared coefficients:

$J_{\text{ridge}}(\theta) = \frac{1}{n}\|y - X\theta\|^2 + \lambda \sum_{j=1}^{d} \theta_j^2$

Note that the intercept $\theta_0$ is conventionally excluded from the penalty, since it should not be shrunk toward zero (doing so would bias predictions toward zero regardless of feature values).

8.2 Closed-Form Solution#

Ridge regression retains a closed-form solution, and critically, it remains well-defined even when $X^T X$ alone is not invertible:

$\hat{\theta}_{\text{ridge}} = (X^T X + \lambda I)^{-1} X^T y$

Adding $\lambda I$ (with $\lambda > 0$) to $X^T X$ guarantees invertibility, which is precisely why Ridge is a practical fix for both multicollinearity and the $d > n$ case described in Section 7.1.

8.3 Effect on Coefficients#

Ridge shrinks all coefficients toward zero, proportionally, but essentially never sets them to exactly zero (except in the limiting case of infinite $\lambda$). This makes Ridge well-suited when you believe most or all features contribute at least a little to the prediction, and you want to reduce variance without discarding any feature entirely.

8.4 Code Example#

Python
CopyAsk AI
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Feature scaling (Notebook 2, Section 4) is essential before regularization,
# since the penalty term treats all coefficients on equal footing regardless
# of the original feature scale.
ridge_model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
ridge_model.fit(X_train, y_train)

print(f"Ridge coefficients: {ridge_model.named_steps['ridge'].coef_}")

Note: in scikit-learn, the regularization strength is called alpha, not lambda (since lambda is a reserved keyword in Python).

---

9. Lasso Regression (L1 Regularization)#

9.1 The Formula#

Lasso (Least Absolute Shrinkage and Selection Operator) adds a penalty proportional to the sum of absolute values of coefficients:

$J_{\text{lasso}}(\theta) = \frac{1}{n}\|y - X\theta\|^2 + \lambda \sum_{j=1}^{d} |\theta_j|$

9.2 No Closed-Form Solution#

Unlike Ridge, the absolute value penalty is not differentiable at zero, so Lasso has no closed-form solution and requires iterative optimization methods (such as coordinate descent, which scikit-learn uses internally).

9.3 Effect on Coefficients: Automatic Feature Selection#

The critical practical difference from Ridge is that Lasso can shrink coefficients to exactly zero, effectively removing those features from the model entirely. This happens because of the geometry of the L1 penalty (its constraint region has sharp corners along the coordinate axes, where the optimal solution often lands exactly), whereas the L2 penalty's smooth, circular constraint region rarely produces an exact-zero solution.

This makes Lasso an embedded feature selection method, directly connecting back to Notebook 2, Section 7.4.

9.4 Code Example#

Python
CopyAsk AI
from sklearn.linear_model import Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

lasso_model = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
lasso_model.fit(X_train, y_train)

coefs = lasso_model.named_steps["lasso"].coef_
n_zero = (coefs == 0).sum()
print(f"Number of features eliminated (coefficient = 0): {n_zero} out of {len(coefs)}")

9.5 When to Prefer Lasso Over Ridge#

Prefer Lasso when you suspect many features are truly irrelevant and want an interpretable, sparse model. Prefer Ridge when you believe most features carry at least some signal and you mainly want to control variance, or when features are highly correlated with each other (Lasso tends to arbitrarily pick one feature from a correlated group and zero out the rest, which can be undesirable if you want to retain all correlated signals).

---

10. Elastic Net#

10.1 The Formula#

Elastic Net combines both penalties, controlled by a mixing parameter $\rho \in [0, 1]$:

$J_{\text{elastic net}}(\theta) = \frac{1}{n}\|y - X\theta\|^2 + \lambda \left[\rho \sum_{j=1}^{d} |\theta_j| + (1-\rho) \sum_{j=1}^{d} \theta_j^2 \right]$

When $\rho = 1$, this reduces exactly to Lasso; when $\rho = 0$, it reduces exactly to Ridge.

10.2 Why Use Elastic Net#

Elastic Net addresses a specific weakness of Lasso: when features are highly correlated, Lasso tends to select only one from the group somewhat arbitrarily. Elastic Net's L2 component encourages correlated features to be selected (or shrunk) together as a group, retaining more of Ridge's stability while still allowing some coefficients to reach exactly zero.

10.3 Code Example#

Python
CopyAsk AI
from sklearn.linear_model import ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

elastic_model = make_pipeline(
    StandardScaler(),
    ElasticNet(alpha=0.1, l1_ratio=0.5)  # l1_ratio corresponds to rho above
)
elastic_model.fit(X_train, y_train)

---

11. Hands-On: Comparing OLS, Ridge, and Lasso#

This example creates a dataset with more features than are truly useful, plus multicollinearity, and compares how each method handles it.

Python
CopyAsk AI
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# 20 features, but only 5 are actually informative; the rest are noise
X, y = make_regression(
    n_samples=100, n_features=20, n_informative=5, noise=10, random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

models = {
    "OLS": LinearRegression(),
    "Ridge (alpha=1.0)": Ridge(alpha=1.0),
    "Lasso (alpha=1.0)": Lasso(alpha=1.0)
}

print(f"{'Model':<20} | {'Test MSE':>10} | {'Nonzero Coefs':>14}")
print("-" * 50)

for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    preds = model.predict(X_test_scaled)
    mse = mean_squared_error(y_test, preds)
    nonzero = np.sum(np.abs(model.coef_) > 1e-6)
    print(f"{name:<20} | {mse:>10.2f} | {nonzero:>14}")

Expected output (approximate, will vary by environment):

code
Ask AICopy
Model                |   Test MSE | Nonzero Coefs
--------------------------------------------------
OLS                  |     412.35 |             20
Ridge (alpha=1.0)    |     301.87 |             20
Lasso (alpha=1.0)    |     198.42 |              6

Interpretation: OLS uses all 20 features, including the 15 pure-noise ones, and suffers the highest test error from fitting that noise. Ridge shrinks all coefficients and improves generalization somewhat, but still uses all 20 features. Lasso correctly identifies that only a handful of features matter, zeroing out most of the noise features, and achieves the best test error in this scenario, directly demonstrating the embedded feature selection property from Section 9.3.

---

12. Case Studies#

12.1 Case Study: Multicollinearity in Economic Forecasting#

An economics team builds a linear regression model to predict consumer spending using GDP, disposable income, and unemployment rate as features. The coefficient on disposable income is negative, contradicting well-established economic theory that higher disposable income increases spending.

Diagnosis: GDP and disposable income are highly correlated (both tend to rise and fall together over the business cycle). Checking VIF (Section 6.2) reveals values above 15 for both features. Per Section 5's multicollinearity assumption, this makes individual coefficients unstable and uninterpretable, even though the model's overall predictions may still be reasonably accurate.

Lesson: A model with strong predictive accuracy can still have individually meaningless or misleading coefficients under high multicollinearity. Ridge regression (Section 8), which is specifically robust to multicollinearity, or removing/combining correlated features, are both appropriate fixes depending on whether the goal is prediction or interpretation.

12.2 Case Study: Lasso for Genomic Feature Selection#

In genomics research, a dataset might have gene expression measurements for 20,000 genes but only a few hundred patient samples, a severe case of the $d > n$ problem from Section 7.1. Researchers use Lasso regression to predict a disease outcome from gene expression levels.

Diagnosis: With $d \gg n$, OLS has no unique solution at all (Section 7.1). Lasso's ability to drive most of the 20,000 coefficients to exactly zero (Section 9.3) both makes the problem solvable and produces a small, interpretable list of candidate genes that domain experts can investigate further.

Lesson: Lasso's sparsity property is not just a performance tool but often a critical scientific communication tool, translating a mathematically intractable problem into a short, actionable list of implicated variables.

12.3 Case Study: Heteroscedasticity in Insurance Claim Prediction#

An insurance company models claim amounts using a linear regression on policyholder attributes. Residual plots (Section 6.1) show a clear funnel shape: residual variance increases sharply for policyholders with higher predicted claim amounts.

Diagnosis: This is a textbook heteroscedasticity violation (Section 5). The homoscedasticity assumption failing means that, although coefficient point estimates remain roughly unbiased, any confidence intervals or hypothesis tests derived from standard OLS formulas would be unreliable, which matters significantly for a regulated industry that needs statistically defensible model outputs.

Lesson: The team addresses this by modeling the log of claim amount instead of the raw amount (a common variance-stabilizing transformation for right-skewed, heteroscedastic financial data), which is checked afterward with a new residual plot to confirm the funnel pattern has been resolved.

---

13. Common Misconceptions and Pitfalls#

---

14. Glossary#

---

15. Exercises#

Exercise 1 (Conceptual)#

A model predicting salary from years of experience and education level produces a negative coefficient on years of experience, which seems counterintuitive. Give two different possible explanations rooted in the concepts from this notebook.

<details> <summary>Solution</summary>

Two plausible explanations: (1) Multicollinearity (Section 5, 6.2, and the economic forecasting case study in Section 12.1) — years of experience may be highly correlated with education level or age in this dataset, making individual coefficients unstable and potentially sign-flipped even if the joint effect of the correlated features is sensible. (2) Omitted relevant interaction or nonlinearity — if the true relationship is nonlinear (e.g., diminishing or even reversing returns to experience at very senior, specialized roles not well represented in the data) and only a linear term is included, the fitted coefficient can reflect a poor linear approximation to a more complex pattern (a linearity assumption violation, Section 5).

</details>

---

Exercise 2 (Conceptual)#

Explain, using the geometry described in Section 9.3, why Lasso can produce exact zero coefficients while Ridge essentially never does.

<details> <summary>Solution</summary>

Both methods can be viewed as minimizing the original loss subject to a constraint on the size of the coefficient vector: Ridge constrains $\sum \theta_j^2 \leq c$ (a circular/spherical region), while Lasso constrains $\sum |\theta_j| \leq c$ (a diamond-shaped region with sharp corners located exactly on the coordinate axes, where one or more coefficients equal zero). Because the loss function's contours are more likely to first touch the constraint region exactly at one of these sharp corners for Lasso, solutions frequently land with some coordinates exactly zero. Ridge's smooth, corner-free spherical boundary makes such an exact-zero intersection point far less likely, so coefficients shrink toward zero smoothly without usually reaching it exactly.

</details>

---

Exercise 3 (Coding)#

Using the diabetes dataset (sklearn.datasets.load_diabetes), fit Ridge models with alpha values of [0.01, 0.1, 1, 10, 100] using 5-fold cross-validation, and identify which alpha minimizes mean cross-validated MSE.

<details> <summary>Solution</summary>

Python
CopyAsk AI
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np

X, y = load_diabetes(return_X_y=True)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)

for alpha in [0.01, 0.1, 1, 10, 100]:
    model = make_pipeline(StandardScaler(), Ridge(alpha=alpha))
    scores = -cross_val_score(model, X, y, cv=kfold, scoring="neg_mean_squared_error")
    print(f"alpha={alpha:>6}: mean MSE={scores.mean():.1f}")

The alpha that minimizes mean cross-validated MSE represents the best bias-variance tradeoff point (Notebook 1, Section 4) for this dataset and feature set; this same cross-validated search procedure, automated, is the subject of Notebook 12 (Hyperparameter Tuning).

</details>

---

Exercise 4 (Applied Thinking)#

You are modeling website conversion rate using 200 highly correlated marketing spend features across different channels (e.g., search, display, social, each further split by dozens of sub-campaigns). Which regularization method would you choose as a starting point, and why?

<details> <summary>Solution</summary>

Elastic Net (Section 10) is a strong starting point here. Pure Lasso (Section 9.5) risks arbitrarily selecting only one feature from each group of highly correlated campaign-level spend features and zeroing out the rest, which would be misleading for a marketing team trying to understand which channels genuinely matter, since correlated channels might jointly represent a real, shared budget-driven pattern. Pure Ridge (Section 8.3) would retain all 200 features, providing little help narrowing down which channels to prioritize. Elastic Net's blend allows correlated groups of features to be shrunk or selected together, providing both dimensionality reduction and more interpretable groupings.

</details>

---

16. Quiz#

Q1. What loss function does Ordinary Least Squares minimize? a) Absolute error b) Squared error c) 0/1 loss d) Log loss

Q2. Which condition makes the Normal Equation's closed-form solution impossible to compute directly? a) The target variable is not normally distributed b) $X^TX$ is non-invertible, such as when there are more features than observations c) The learning rate is too high d) The residuals are heteroscedastic

Q3. Which regularization method can produce coefficients that are exactly zero? a) Ridge only b) Lasso only c) Both Ridge and Lasso d) Neither Ridge nor Lasso

Q4. A residual plot shows a clear funnel shape, with variance increasing as predicted values increase. Which OLS assumption is most directly being violated? a) Linearity b) Independence c) Homoscedasticity d) No multicollinearity

Q5. True or False: Regularization should always be applied after scaling features, not before.

<details> <summary>Answers</summary>

1. b) Squared error
2. b) $X^TX$ is non-invertible, such as when there are more features than observations
3. b) Lasso only, due to the geometry described in Section 9.3
4. c) Homoscedasticity
5. False as phrased in terms of ordering — scaling should be applied, and regularization is then applied to the already-scaled features. In pipeline terms this means scaling happens first, then regularized model fitting happens on the scaled output; the statement is essentially correct in spirit (scale before regularizing) but the notebook frames it as "scaling before fitting" rather than a strict temporal "after" — the key takeaway from Section 8.4 and 9.4 is that features must be scaled prior to fitting a regularized model, which several quiz-takers may reasonably read either way; the important fact to retain is that unscaled features and regularization should not be combined.

</details>

---

17. Summary#

• Linear regression models the target as a weighted sum of features, with weights found via Ordinary Least Squares, which minimizes the sum of squared residuals.
• OLS has a closed-form solution (the Normal Equation) for simple and multiple regression, but this solution requires $X^TX$ to be invertible.
• Five classical assumptions (linearity, independence, homoscedasticity, normal residuals, low multicollinearity) determine when OLS coefficient interpretations and confidence intervals can be trusted, and residual plots and VIF are the primary diagnostic tools.
• Regularization (Ridge, Lasso, Elastic Net) addresses overfitting and multicollinearity by penalizing large coefficients, trading a small increase in bias for a often much larger reduction in variance.
• Ridge shrinks coefficients smoothly toward zero without eliminating them; Lasso can drive coefficients to exactly zero, providing embedded feature selection; Elastic Net blends both behaviors, particularly useful with correlated feature groups.
• Feature scaling is a prerequisite for meaningful, fair regularization across features of different original scales.

---

18. Further Reading#

• Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements of Statistical Learning, Chapter 3 (Linear Methods for Regression)
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning, Chapter 3 and Chapter 6
• scikit-learn documentation on linear models: https://scikit-learn.org/stable/modules/linear_model.html
• Robert Tibshirani's original 1996 paper introducing Lasso, "Regression Shrinkage and Selection via the Lasso"

---

Next Up#

Notebook 4: Logistic Regression and Classification Basics — the sigmoid function, decision boundaries, and strategies for extending binary classification to multiple classes, extending the loss function and optimization ideas introduced in this notebook to categorical targets.

---

Course: Machine Learning | Notebook 3 of 13