Hyperparameter Tuning and Model Selection

Overview#

Every algorithm covered in this course has hyperparameters — settings chosen before training, not learned from data (like alpha in Ridge, k in KNN, or max_depth in a decision tree). Choosing them well can be the difference between a mediocre model and a genuinely strong one. This notebook covers grid search, random search, Bayesian optimization, and how to build proper scikit-learn pipelines that make all of this reliable and free of data leakage — pulling together concepts from throughout this entire course.

1. Parameters vs Hyperparameters#

text
Ask AICopy
PARAMETERS: learned automatically FROM the data during training.
  Example: the coefficients in linear regression, the split thresholds
  in a decision tree.

HYPERPARAMETERS: set BEFORE training begins, controlling how the
  learning process itself happens.
  Example: alpha in Ridge/Lasso, k in KNN, max_depth in a decision tree,
  C and gamma in SVM, learning_rate in gradient boosting.

Hyperparameter tuning is the process of searching over different hyperparameter combinations to find the ones that produce the best generalization performance — evaluated via cross-validation, exactly as covered in the Statistical Learning Foundations notebook.

2. Grid Search: Exhaustive Search#

Grid search tries every possible combination of specified hyperparameter values, evaluating each with cross-validation.

Python
CopyAsk AI
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("svm", SVC()),
])

param_grid = {
    "svm__C": [0.1, 1, 10, 100],
    "svm__gamma": [0.001, 0.01, 0.1, 1],
    "svm__kernel": ["rbf", "linear"],
}

grid_search = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,       # use all available CPU cores
    verbose=1,
)
grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")
print(f"Test score: {grid_search.score(X_test, y_test):.3f}")

This grid tries 4 × 4 × 2 = 32 combinations, each evaluated with 5-fold cross-validation — 160 total model fits. Grid search's exhaustiveness is also its main weakness: the number of combinations grows multiplicatively with each additional hyperparameter, quickly becoming computationally infeasible for more than a handful of parameters ("the curse of dimensionality" applied to search spaces, not features).

Inspecting all results, not just the best#

Python
CopyAsk AI
import pandas as pd

results_df = pd.DataFrame(grid_search.cv_results_)
top_results = results_df.sort_values("mean_test_score", ascending=False).head(5)
print(top_results[["params", "mean_test_score", "std_test_score"]])

Checking multiple top results (not just the single best) is worthwhile — if several very different hyperparameter combinations score similarly well, that's often a sign the model is fairly insensitive to that particular choice, while a narrow, sharp peak suggests the model is more sensitive and worth investigating further.

3. Random Search: Sampling Instead of Exhausting#

Random search samples a fixed number of random combinations from specified distributions, rather than trying every single combination.

Python
CopyAsk AI
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform, loguniform

param_distributions = {
    "svm__C": loguniform(1e-2, 1e2),         # samples on a log scale — appropriate since
    "svm__gamma": loguniform(1e-3, 1e1),        # good C/gamma values often span orders of magnitude
    "svm__kernel": ["rbf", "linear"],
}

random_search = RandomizedSearchCV(
    pipeline,
    param_distributions,
    n_iter=50,        # only 50 combinations tried, regardless of how large the space is
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
    random_state=42,
)
random_search.fit(X_train, y_train)

print(f"Best parameters: {random_search.best_params_}")
print(f"Best CV score: {random_search.best_score_:.3f}")

Why random search often outperforms grid search for the same budget#

text
Ask AICopy
Counterintuitive but well-established result: for the SAME number of
total model fits, random search often finds BETTER hyperparameters than
grid search, especially when some hyperparameters matter much more than
others.

Grid search wastes evaluations by exhaustively varying EVERY parameter
across a small, fixed set of values — including parameters that barely
matter, and only sparsely covering parameters that matter a lot.

Random search, sampling from continuous distributions, explores a much
WIDER range of values for every parameter, increasing the chance of
landing near a good value for the parameters that actually matter most —
even if you don't know in advance which ones those are.

Architecture & Data Flow
Copy
Visual intuition (2 hyperparameters, one important, one not):

Grid search (3x3 = 9 points):        Random search (9 points):
important                             important
  │ x  x  x                            │  x      x
  │ x  x  x        ← only 3 distinct   │      x
  │ x  x  x           values tried     │ x  x       x  ← 9 distinct
  └──────── unimportant                └──────── unimportant  values tried
                                                    for "important"

4. Bayesian Optimization: Smarter, Sequential Search#

Both grid and random search choose all combinations upfront, independent of each other's results. Bayesian optimization is smarter: it uses the results of previous evaluations to decide which combination to try next, focusing the search on the most promising regions.

bash
Ask AICopy
pip install scikit-optimize --break-system-packages

Python
CopyAsk AI
from skopt import BayesSearchCV
from skopt.space import Real, Categorical

bayes_search = BayesSearchCV(
    pipeline,
    {
        "svm__C": Real(1e-2, 1e2, prior="log-uniform"),
        "svm__gamma": Real(1e-3, 1e1, prior="log-uniform"),
        "svm__kernel": Categorical(["rbf", "linear"]),
    },
    n_iter=30,       # fewer iterations often needed than random search, since it's smarter
    cv=5,
    scoring="accuracy",
    random_state=42,
)
bayes_search.fit(X_train, y_train)

print(f"Best parameters: {bayes_search.best_params_}")
print(f"Best CV score: {bayes_search.best_score_:.3f}")

text
Ask AICopy
How it works, conceptually:
1. Evaluate a few initial random hyperparameter combinations
2. Build a probabilistic model (often a Gaussian Process) predicting how
   the score varies across the hyperparameter space, based on results
   observed SO FAR
3. Use that model to pick the NEXT combination to try — balancing
   "exploitation" (trying values near the best-known result) against
   "exploration" (trying uncertain regions that might hide something
   better)
4. Repeat, refining the probabilistic model with each new result

Bayesian optimization typically finds strong hyperparameters with fewer total evaluations than random search, which matters most when each individual model fit is slow/expensive (e.g. deep learning, or gradient boosting on very large data) — for fast-to-train models on small data, the overhead of the smarter search strategy may not be worth it over simple random search.

5. Choosing a Search Strategy#

6. Nested Cross-Validation: Avoiding an Overly Optimistic Estimate#

If you use cross-validation to both select hyperparameters AND estimate final model performance using the same data, the reported score tends to be optimistically biased — you've effectively tuned the model to perform well on that specific validation data.

Python
CopyAsk AI
from sklearn.model_selection import cross_val_score, KFold

# Nested cross-validation: an OUTER loop estimates generalization
# performance, while an INNER loop (inside each outer fold) handles
# hyperparameter selection independently
inner_cv = KFold(n_splits=5, shuffle=True, random_state=1)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=2)

grid_search_inner = GridSearchCV(pipeline, param_grid, cv=inner_cv, scoring="accuracy")

nested_scores = cross_val_score(grid_search_inner, X_train, y_train, cv=outer_cv)
print(f"Nested CV score: {nested_scores.mean():.3f} (+/- {nested_scores.std():.3f})")

This gives a more honest, less optimistically biased estimate of how the whole process — hyperparameter selection AND model fitting together — will generalize to genuinely new data. It's more computationally expensive (an entire grid search runs inside every outer fold), so it's typically reserved for final, rigorous performance reporting rather than routine day-to-day tuning.

7. Building Robust Pipelines#

Correctly combining preprocessing and modeling into a single Pipeline is what makes all of the above trustworthy — without it, cross-validation and hyperparameter search results can be subtly (or badly) wrong due to data leakage.

Python
CopyAsk AI
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

numeric_features = ["age", "income"]
categorical_features = ["occupation", "region"]

numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features),
])

full_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(random_state=42)),
])

# Now this ENTIRE pipeline — imputation, scaling, encoding, AND modeling —
# can be cross-validated and hyperparameter-tuned as one unit
param_grid = {
    "classifier__n_estimators": [100, 200],
    "classifier__max_depth": [5, 10, None],
}
grid_search = GridSearchCV(full_pipeline, param_grid, cv=5)
grid_search.fit(X_train, y_train)

ColumnTransformer applies different preprocessing to different columns (numeric vs categorical) within a single unified pipeline — this is the standard, production-grade pattern for real-world datasets that mix feature types, ensuring every preprocessing step is refit correctly within each cross-validation fold, with zero leakage between them.

8. Comparing Multiple Model Types#

Python
CopyAsk AI
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000),
    "Random Forest": RandomForestClassifier(random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(random_state=42),
    "SVM": SVC(),
}

for name, model in models.items():
    pipeline = Pipeline([("scaler", StandardScaler()), ("model", model)])
    scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="accuracy")
    print(f"{name}: {scores.mean():.3f} (+/- {scores.std():.3f})")

Comparing several genuinely different model types via cross-validation, before investing heavily in tuning any single one, is good practice — the "No Free Lunch" principle (covered in the Statistical Learning Foundations notebook) means no algorithm is guaranteed best for a given problem, so empirical comparison remains essential.

9. Practical Tuning Workflow#

text
Ask AICopy
1. Establish a baseline with default hyperparameters across a few
   different model types (as in section 8)
2. Pick the most promising 1-2 model types based on that baseline
3. Use random search (or Bayesian optimization for expensive models) to
   broadly explore the hyperparameter space for those models
4. Narrow to a focused grid search around the best region found by the
   broad search, if further refinement seems worthwhile
5. Validate the final choice with nested cross-validation (or a final,
   untouched test set) for an honest performance estimate
6. Only then, evaluate on the true held-out test set — exactly once

Notes#

• Hyperparameters are set before training; parameters are learned during training — tuning refers specifically to searching over the former.
• Grid search is exhaustive but scales poorly with additional hyperparameters; random search often finds comparably good or better results with the same computational budget, especially when only a few hyperparameters actually matter much.
• Bayesian optimization is most valuable when individual model fits are expensive — the smarter search strategy's overhead isn't worth it for fast, cheap models.
• Using the same cross-validation results for both hyperparameter selection and final performance reporting produces an optimistically biased estimate — nested cross-validation (or a genuinely untouched test set) avoids this.
• ColumnTransformer combined with Pipeline is the standard, leakage-free way to handle mixed numeric/categorical preprocessing alongside hyperparameter tuning.

Best Practices Checklist#

• Wrap all preprocessing steps in a Pipeline (with ColumnTransformer for mixed feature types) before any cross-validation or tuning
• Use grid search only for a small number of hyperparameters with a well-understood range
• Prefer random search over grid search when tuning several hyperparameters simultaneously, for the same computational budget
• Reach for Bayesian optimization when individual model fits are expensive enough that minimizing total evaluations matters
• Inspect multiple top-scoring hyperparameter combinations, not just the single best, to gauge sensitivity
• Use nested cross-validation (or a truly untouched test set) for final, honest performance reporting
• Compare several different model types with cross-validation before investing heavily in tuning any single one
• Never use the same cross-validation results for both hyperparameter selection and final performance claims