Regularization Techniques

Overview#

Regularization is the primary tool for controlling overfitting: it penalizes model complexity directly, discouraging a model from fitting noise in the training data. This notebook covers Ridge (L2) regression, Lasso (L1) regression, Elastic Net (a combination of both), how regularization strength connects directly to the bias-variance tradeoff covered in the Statistical Learning Foundations notebook, and practical guidance on choosing between them.

1. Why Regularization Is Needed#

A standard linear regression minimizes the sum of squared errors on the training data, with no penalty for how large or extreme the model's coefficients become. With many features (especially correlated ones) or limited data, this can lead to very large coefficients that fit the training data closely but generalize poorly — classic overfitting, high variance.

Python
CopyAsk AI
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

np.random.seed(42)
X = np.sort(np.random.rand(20, 1) * 10, axis=0)
y = 2 * X.ravel() + 1 + np.random.randn(20) * 2

# A high-degree polynomial on limited data — prone to wild, overfit coefficients
model = make_pipeline(PolynomialFeatures(10), LinearRegression())
model.fit(X, y)

coefficients = model.named_steps["linearregression"].coef_
print(f"Coefficient magnitude range: {coefficients.min():.1f} to {coefficients.max():.1f}")
# Coefficients can become extremely large — a hallmark of an overfit model

Regularization directly discourages large coefficients by adding a penalty term to the loss function being minimized, trading a small increase in training error for (hopefully) a large decrease in test error — deliberately introducing bias to reduce variance.

2. Ridge Regression (L2 Regularization)#

Ridge regression adds a penalty proportional to the sum of squared coefficients to the ordinary least squares loss function.

Mathematical Formulation
Explain FormulaCopy
Ridge loss = Sum of squared errors + alpha * Sum(coefficient²)

Python
CopyAsk AI
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=200, n_features=20, n_informative=10, noise=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scaling is essential before regularization — see the note below
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

ridge = Ridge(alpha=1.0)   # alpha controls regularization strength
ridge.fit(X_train_scaled, y_train)

print(f"Test R²: {ridge.score(X_test_scaled, y_test):.3f}")
print(f"Coefficient magnitudes: {np.abs(ridge.coef_).mean():.2f}")

The effect of alpha on coefficients#

Python
CopyAsk AI
alphas = [0, 0.1, 1, 10, 100]
for a in alphas:
    ridge = Ridge(alpha=a)
    ridge.fit(X_train_scaled, y_train)
    print(f"alpha={a:>5}: mean |coef| = {np.abs(ridge.coef_).mean():.3f}, "
          f"test R² = {ridge.score(X_test_scaled, y_test):.3f}")

text
Ask AICopy
alpha=    0: mean |coef| = 42.130, test R² = 0.812
alpha=  0.1: mean |coef| = 41.982, test R² = 0.815
alpha=    1: mean |coef| = 40.671, test R² = 0.821
alpha=   10: mean |coef| = 32.104, test R² = 0.834
alpha=  100: mean |coef| = 14.223, test R² = 0.791

Increasing alpha shrinks coefficients toward zero (but rarely to exactly zero) — this is Ridge's key characteristic. Notice test R² often improves initially as alpha increases (reducing overfitting), but eventually decreases if alpha becomes too large (the model becomes too constrained — underfitting).

Key property: Ridge shrinks coefficients but rarely to exactly zero#

Python
CopyAsk AI
ridge = Ridge(alpha=50)
ridge.fit(X_train_scaled, y_train)
print(f"Number of exactly-zero coefficients: {np.sum(ridge.coef_ == 0)}")
# Typically 0 — Ridge shrinks ALL coefficients proportionally, it doesn't
# eliminate features entirely

3. Lasso Regression (L1 Regularization)#

Lasso adds a penalty proportional to the sum of absolute values of the coefficients, instead of their squares.

Mathematical Formulation
Explain FormulaCopy
Lasso loss = Sum of squared errors + alpha * Sum(|coefficient|)

Python
CopyAsk AI
from sklearn.linear_model import Lasso

lasso = Lasso(alpha=1.0)
lasso.fit(X_train_scaled, y_train)

print(f"Test R²: {lasso.score(X_test_scaled, y_test):.3f}")
print(f"Number of exactly-zero coefficients: {np.sum(lasso.coef_ == 0)} out of {len(lasso.coef_)}")

›Test R²: 0.798
Number of exactly-zero coefficients: 8 out of 20

Lasso's key property: automatic feature selection#

Unlike Ridge, Lasso can shrink coefficients to exactly zero, effectively removing those features from the model entirely. This makes Lasso useful not just for regularization, but as an automatic feature selection method.

Python
CopyAsk AI
alphas = [0.01, 0.1, 1, 10]
for a in alphas:
    lasso = Lasso(alpha=a, max_iter=10000)
    lasso.fit(X_train_scaled, y_train)
    n_zero = np.sum(lasso.coef_ == 0)
    print(f"alpha={a:>5}: {n_zero}/20 coefficients are zero, test R² = {lasso.score(X_test_scaled, y_test):.3f}")

text
Ask AICopy
alpha= 0.01: 0/20 coefficients are zero, test R² = 0.819
alpha=  0.1: 2/20 coefficients are zero, test R² = 0.821
alpha=    1: 8/20 coefficients are zero, test R² = 0.798
alpha=   10: 18/20 coefficients are zero, test R² = 0.412

As alpha increases, more coefficients get pushed to exactly zero — at very high alpha, nearly everything is eliminated and the model underfits badly, having lost almost all its predictive features.

Why Lasso can zero out coefficients but Ridge can't (geometric intuition)#

text
Ask AICopy
Ridge's penalty region (sum of squares) is a smooth circle/sphere —
the optimal solution rarely lands exactly on an axis.

Lasso's penalty region (sum of absolute values) is a diamond/polyhedron
with sharp CORNERS on the axes — the optimal solution frequently lands
exactly on a corner, where one or more coefficients are exactly zero.

This geometric difference is why L1 regularization produces SPARSE
solutions (many exact zeros) while L2 regularization produces DENSE
solutions (small but nonzero values).

4. Elastic Net: Combining Both#

Elastic Net combines the L1 and L2 penalties, controlled by an additional l1_ratio parameter that balances between them.

Mathematical Formulation
Explain FormulaCopy
Elastic Net loss = Sum of squared errors
                    + alpha * [l1_ratio * Sum(|coef|) + (1 - l1_ratio) * Sum(coef²)]

Python
CopyAsk AI
from sklearn.linear_model import ElasticNet

elastic = ElasticNet(alpha=1.0, l1_ratio=0.5)   # 0.5 = equal mix of L1 and L2
elastic.fit(X_train_scaled, y_train)

print(f"Test R²: {elastic.score(X_test_scaled, y_test):.3f}")
print(f"Number of exactly-zero coefficients: {np.sum(elastic.coef_ == 0)}")

• l1_ratio=1.0 → pure Lasso
• l1_ratio=0.0 → pure Ridge
• l1_ratio between 0 and 1 → a blend of both

When Elastic Net helps over pure Lasso#

Lasso has a known limitation: when several features are highly correlated, it tends to arbitrarily pick just one of them and zero out the rest, which can be unstable (a slightly different sample might cause it to pick a different one from the correlated group). Elastic Net's L2 component encourages correlated features to be selected (or shrunk) together, producing more stable results in this situation.

Python
CopyAsk AI
# Simulating correlated features
X_corr = np.random.randn(200, 3)
X_corr = np.column_stack([X_corr, X_corr[:, 0] + np.random.randn(200) * 0.01])  # near-duplicate of feature 0
y_corr = 3 * X_corr[:, 0] + np.random.randn(200)

lasso = Lasso(alpha=0.5).fit(X_corr, y_corr)
elastic = ElasticNet(alpha=0.5, l1_ratio=0.5).fit(X_corr, y_corr)

print(f"Lasso coefficients:       {lasso.coef_}")     # likely zeros out one of the correlated pair
print(f"Elastic Net coefficients: {elastic.coef_}")    # tends to share weight between them

5. Regularization and the Bias-Variance Tradeoff#

Regularization strength (alpha) directly controls where a model sits on the bias-variance spectrum covered in the Statistical Learning Foundations notebook.

Python
CopyAsk AI
from sklearn.model_selection import cross_val_score

alphas = np.logspace(-3, 3, 20)   # 0.001 to 1000, log-spaced
train_scores = []
cv_scores = []

for a in alphas:
    ridge = Ridge(alpha=a)
    ridge.fit(X_train_scaled, y_train)
    train_scores.append(ridge.score(X_train_scaled, y_train))
    cv_scores.append(cross_val_score(ridge, X_train_scaled, y_train, cv=5).mean())

# Plotting train_scores vs cv_scores against alpha reveals the classic
# bias-variance curve:
#   - Low alpha: high train score, lower cv score -> overfitting (high variance)
#   - High alpha: both scores converge and drop -> underfitting (high bias)
#   - Some middle alpha: cv score peaks -> the best bias-variance balance

text
Ask AICopy
alpha=0.001: train R²=0.94, cv R²=0.71   <- overfitting: big gap
alpha=1.0:   train R²=0.89, cv R²=0.83   <- reasonable balance
alpha=100:   train R²=0.65, cv R²=0.64   <- underfitting: both low

The growing gap between training and cross-validation score at low alpha is the signature of high variance (overfitting); both scores dropping together at high alpha is the signature of high bias (underfitting). The best alpha is the one that maximizes cross- validation score, not training score.

6. Choosing alpha with Cross-Validation#

Python
CopyAsk AI
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV

# Automatically searches over a range of alpha values using cross-validation
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 50), cv=5)
ridge_cv.fit(X_train_scaled, y_train)
print(f"Best alpha (Ridge): {ridge_cv.alpha_:.3f}")

lasso_cv = LassoCV(alphas=np.logspace(-3, 1, 50), cv=5, max_iter=10000)
lasso_cv.fit(X_train_scaled, y_train)
print(f"Best alpha (Lasso): {lasso_cv.alpha_:.3f}")

elastic_cv = ElasticNetCV(l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9, 1.0], cv=5, max_iter=10000)
elastic_cv.fit(X_train_scaled, y_train)
print(f"Best alpha: {elastic_cv.alpha_:.3f}, best l1_ratio: {elastic_cv.l1_ratio_:.2f}")

RidgeCV/LassoCV/ElasticNetCV handle the alpha search internally using efficient cross-validation, rather than needing a separate manual grid search loop — the recommended way to select regularization strength in practice.

7. Feature Scaling Is Essential#

Python
CopyAsk AI
# WITHOUT scaling — features on different scales get penalized unfairly
X_unscaled_example = np.array([[1, 1000], [2, 2000], [3, 3000]])
# A coefficient of 0.001 on the second feature and 1.0 on the first could
# represent EQUALLY important relationships, but the regularization
# penalty would punish the second feature's coefficient far more,
# purely due to the different scale of the raw feature.

# WITH scaling — every feature is on a comparable scale, so the
# regularization penalty treats them fairly based on actual importance
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_unscaled_example)

Regularization penalizes coefficient magnitude directly, so features on different scales get penalized unequally unless they're first scaled to comparable ranges — always scale features (typically with StandardScaler) before applying Ridge, Lasso, or Elastic Net.

8. Summary Comparison#

Notes#

• Regularization deliberately trades a small increase in bias for a (hopefully larger) decrease in variance — it's a tool for managing the bias-variance tradeoff directly, not a way to "fix" a model for free.
• Always scale features before applying any of these three methods — unscaled features make the regularization penalty unfairly favor naturally small-magnitude features.
• Ridge never fully eliminates a feature (coefficients shrink but stay nonzero); Lasso can eliminate features entirely, effectively performing feature selection as a side effect of regularization.
• Lasso's feature selection can be unstable with highly correlated features — Elastic Net's added L2 component generally produces more stable, reproducible results in that situation.
• RidgeCV, LassoCV, and ElasticNetCV are the standard, efficient way to select alpha (and l1_ratio for Elastic Net) via cross-validation, rather than a manual search loop.

Best Practices Checklist#

• Always scale features (e.g. with StandardScaler) before applying Ridge, Lasso, or Elastic Net
• Use cross-validation (RidgeCV/LassoCV/ElasticNetCV) to select alpha, rather than picking it arbitrarily
• Choose Lasso when you suspect many features are irrelevant and want automatic feature selection
• Choose Ridge when most features are likely at least somewhat useful and you want to keep them all, just with controlled magnitude
• Choose Elastic Net when features are correlated but sparsity is still desired
• Watch for the gap between training and cross-validation score to diagnose whether alpha is too low (overfitting) or too high (underfitting)
• Don't interpret Lasso's selected/zeroed features as a definitive causal statement — correlated features can cause instability in which ones get selected
• Remember regularization is one tool among several for controlling model complexity, not a universal fix for all overfitting problems