Scikit-learn Deep Dive and End-to-End ML Workflow

Overview#

This final notebook ties together everything from the Machine Learning course into one coherent, realistic project workflow — from raw data to a deployed, saved model. It covers scikit-learn's estimator API in depth, writing custom transformers, building a complete ColumnTransformer + Pipeline for mixed data, integrating everything from preprocessing through hyperparameter tuning, evaluation, and model persistence for later use in a real application.

1. The Scikit-learn Estimator API#

Every scikit-learn model, transformer, and preprocessing step follows the same consistent interface, which is why pipelines work at all.

Python
CopyAsk AI
# Every ESTIMATOR (model) implements:
model.fit(X, y)              # learn from training data
model.predict(X)               # make predictions on new data
model.score(X, y)                # evaluate performance (accuracy/R² by default)

# Every TRANSFORMER (preprocessing step) implements:
transformer.fit(X)             # learn parameters from training data (e.g. mean/std for scaling)
transformer.transform(X)         # apply the learned transformation
transformer.fit_transform(X)       # convenience: fit then transform in one call

This consistency is what makes Pipeline possible — scikit-learn can chain any sequence of transformers followed by a final estimator, because they all speak the same fit/transform/predict language, regardless of what each one actually does internally.

2. Writing a Custom Transformer#

Real projects frequently need preprocessing logic that scikit-learn doesn't provide out of the box. Writing a custom transformer lets that logic participate fully in pipelines, cross-validation, and hyperparameter search, exactly like a built-in one.

Python
CopyAsk AI
from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class OutlierClipper(BaseEstimator, TransformerMixin):
    """Clips values beyond a given number of standard deviations from the mean."""

    def __init__(self, n_std=3.0):
        self.n_std = n_std   # hyperparameter, must match constructor argument name

    def fit(self, X, y=None):
        X = np.asarray(X)
        self.mean_ = X.mean(axis=0)
        self.std_ = X.std(axis=0)
        return self   # fit() must always return self

    def transform(self, X):
        X = np.asarray(X)
        lower = self.mean_ - self.n_std * self.std_
        upper = self.mean_ + self.n_std * self.std_
        return np.clip(X, lower, upper)


clipper = OutlierClipper(n_std=2.5)
X_clipped = clipper.fit_transform(X_train)

Rules for a correct custom transformer#

text
Ask AICopy
1. Inherit from BaseEstimator and TransformerMixin — this gives you
   get_params()/set_params() for free (needed by GridSearchCV) and
   fit_transform() for free (from fit + transform).
2. __init__ should ONLY store its arguments as attributes with matching
   names — no logic, no validation, no computation there.
3. fit() learns anything needed FROM the training data (means, std devs,
   categories seen, etc.) and stores it with a trailing underscore
   (self.mean_, not self.mean) — this is scikit-learn's convention for
   "attributes learned during fit".
4. fit() must return self.
5. transform() applies the learned transformation — it should NEVER learn
   anything new from the data it's given (that would cause data leakage
   when used on validation/test data).

A custom feature-engineering transformer#

Python
CopyAsk AI
class RatioFeatureAdder(BaseEstimator, TransformerMixin):
    """Adds a new feature: the ratio between two existing columns."""

    def __init__(self, numerator_idx, denominator_idx):
        self.numerator_idx = numerator_idx
        self.denominator_idx = denominator_idx

    def fit(self, X, y=None):
        return self   # nothing to learn from the data for this transformer

    def transform(self, X):
        X = np.asarray(X)
        ratio = X[:, self.numerator_idx] / (X[:, self.denominator_idx] + 1e-8)
        return np.column_stack([X, ratio])

Custom transformers like this let domain-specific feature engineering (ratios, interaction terms, business-specific rules) live inside the pipeline itself — meaning it gets applied consistently and correctly during cross-validation, hyperparameter search, and eventual production use, rather than as a separate, easy-to-forget manual step.

3. A Complete End-to-End Project: Predicting Customer Churn#

This section walks through a realistic, complete workflow, pulling together concepts from across the whole course.

Step 1: Load and explore the data#

Python
CopyAsk AI
import pandas as pd
import numpy as np

# Simulating a realistic mixed-type dataset
np.random.seed(42)
n = 1000
df = pd.DataFrame({
    "tenure_months": np.random.randint(1, 72, n),
    "monthly_charges": np.random.uniform(20, 120, n),
    "contract_type": np.random.choice(["month-to-month", "one-year", "two-year"], n),
    "payment_method": np.random.choice(["credit card", "bank transfer", "electronic check"], n),
    "total_charges": None,   # will compute, with some missing values injected below
    "churned": np.random.choice([0, 1], n, p=[0.73, 0.27]),
})
df["total_charges"] = df["tenure_months"] * df["monthly_charges"] + np.random.randn(n) * 50
df.loc[np.random.choice(n, 30, replace=False), "total_charges"] = np.nan   # simulate missing data

print(df.info())
print(df["churned"].value_counts(normalize=True))   # check class balance — relevant per the
                                                        # Model Evaluation notebook

Step 2: Split data BEFORE any preprocessing#

Python
CopyAsk AI
from sklearn.model_selection import train_test_split

X = df.drop(columns=["churned"])
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Splitting before preprocessing (not after) is essential — fitting a scaler, imputer, or encoder on the full dataset before splitting would leak test-set information into training, producing overly optimistic cross-validation and test scores, as covered in the Statistical Learning Foundations notebook.

Step 3: Build the preprocessing pipeline#

Python
CopyAsk AI
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

numeric_features = ["tenure_months", "monthly_charges", "total_charges"]
categorical_features = ["contract_type", "payment_method"]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),   # handles the missing total_charges values
    ("scaler", StandardScaler()),
])

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", drop="first")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numeric_features),
    ("cat", categorical_pipeline, categorical_features),
])

Step 4: Compare multiple model types#

Python
CopyAsk AI
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "Random Forest": RandomForestClassifier(random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(random_state=42),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for name, model in models.items():
    full_pipeline = Pipeline([("preprocessor", preprocessor), ("classifier", model)])
    scores = cross_val_score(full_pipeline, X_train, y_train, cv=cv, scoring="roc_auc")
    print(f"{name}: ROC-AUC = {scores.mean():.3f} (+/- {scores.std():.3f})")

scoring="roc_auc" is chosen here deliberately since churn prediction is a somewhat imbalanced classification problem (per the class distribution checked in Step 1) — matching the metric choice guidance from the Model Evaluation notebook.

Step 5: Tune the best-performing model#

Python
CopyAsk AI
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

best_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier(random_state=42)),
])

param_distributions = {
    "classifier__n_estimators": randint(50, 300),
    "classifier__max_depth": randint(2, 6),
    "classifier__learning_rate": [0.01, 0.05, 0.1, 0.2],
    "classifier__subsample": [0.7, 0.8, 0.9, 1.0],
}

search = RandomizedSearchCV(
    best_pipeline,
    param_distributions,
    n_iter=30,
    cv=cv,
    scoring="roc_auc",
    random_state=42,
    n_jobs=-1,
)
search.fit(X_train, y_train)

print(f"Best parameters: {search.best_params_}")
print(f"Best CV ROC-AUC: {search.best_score_:.3f}")

Step 6: Final evaluation on the untouched test set#

Python
CopyAsk AI
from sklearn.metrics import classification_report, roc_auc_score, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

final_model = search.best_estimator_   # the full pipeline, refit on all training data with best params

y_pred = final_model.predict(X_test)
y_proba = final_model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print(f"Test ROC-AUC: {roc_auc_score(y_test, y_proba):.3f}")

ConfusionMatrixDisplay.from_predictions(y_test, y_pred)
plt.show()

This is the one and only time the test set is touched — per the methodology established in the Statistical Learning Foundations notebook, everything up to this point (model comparison, hyperparameter tuning) used only the training set and cross-validation.

4. Model Persistence: Saving and Loading#

Once a final model is selected, it needs to be saved so it can be used later without retraining — in a web service, batch job, or other application.

Python
CopyAsk AI
import joblib

# Saving the ENTIRE pipeline — preprocessing AND model together
joblib.dump(final_model, "churn_model_pipeline.joblib")

# Loading it back later (e.g. in a completely different script or service)
loaded_model = joblib.load("churn_model_pipeline.joblib")

new_customer = pd.DataFrame({
    "tenure_months": [12],
    "monthly_charges": [65.0],
    "contract_type": ["month-to-month"],
    "payment_method": ["electronic check"],
    "total_charges": [780.0],
})
prediction = loaded_model.predict(new_customer)
probability = loaded_model.predict_proba(new_customer)[:, 1]
print(f"Churn prediction: {prediction[0]}, probability: {probability[0]:.3f}")

joblib is generally preferred over Python's built-in pickle for scikit-learn models — it's more efficient for objects containing large NumPy arrays, which most fitted models are full of.

Critical: saving the WHOLE pipeline, not just the model#

text
Ask AICopy
Saving ONLY the final classifier (without the preprocessing steps) is a
common and serious mistake — the raw new data (categorical strings,
unscaled numbers, missing values) would need to be manually preprocessed
IDENTICALLY to how training data was, every single time, by hand, with
guaranteed eventual drift and bugs.

Saving the FULL PIPELINE (imputation + scaling + encoding + model, all
together, as built in Step 3-5 above) means raw new data can be passed
directly to .predict() and everything happens automatically and
consistently, exactly as it did during training.

5. Versioning and Reproducibility#

Python
CopyAsk AI
import sklearn
import joblib
import json

metadata = {
    "sklearn_version": sklearn.__version__,
    "model_type": type(final_model.named_steps["classifier"]).__name__,
    "best_params": search.best_params_,
    "cv_score": search.best_score_,
    "training_date": "2026-09-16",
    "feature_names": numeric_features + categorical_features,
}

with open("model_metadata.json", "w") as f:
    json.dump(metadata, f, indent=2, default=str)

Recording metadata alongside a saved model — library versions, hyperparameters, training date, performance at training time — is essential for debugging issues later (a model loaded with a mismatched scikit-learn version can behave unpredictably or fail to load at all) and for tracking how a deployed model compares to newer candidates over time.

6. Inspecting Pipeline Internals#

Python
CopyAsk AI
# Accessing named steps directly
print(final_model.named_steps.keys())   # dict_keys(['preprocessor', 'classifier'])

# Getting feature names AFTER preprocessing (including one-hot encoded columns)
feature_names = final_model.named_steps["preprocessor"].get_feature_names_out()
print(feature_names)

# Feature importance, matched up with the actual (post-encoding) feature names
importances = final_model.named_steps["classifier"].feature_importances_
for name, importance in sorted(zip(feature_names, importances), key=lambda x: -x[1])[:5]:
    print(f"{name}: {importance:.3f}")

get_feature_names_out() is invaluable once one-hot encoding is involved — the number and order of features after the ColumnTransformer rarely matches the original input columns directly, and this method reconstructs the correspondence automatically.

7. A Reusable Project Template#

text
Ask AICopy
1. Load data, explore it (shape, types, missing values, class/target
   balance)
2. Split into train/test BEFORE any preprocessing
3. Build preprocessing as a ColumnTransformer (numeric + categorical
   pipelines), adding custom transformers for domain-specific feature
   engineering as needed
4. Compare several model types via cross-validation on the training set,
   using a metric appropriate to the problem (per the Model Evaluation
   notebook)
5. Tune the most promising model(s) with random search or Bayesian
   optimization, wrapped in the SAME pipeline
6. Evaluate the final, tuned pipeline on the untouched test set — exactly
   once
7. Save the full pipeline (not just the model) with joblib, along with
   metadata for reproducibility
8. Load and use the saved pipeline in the target application, passing it
   raw, unprocessed data directly

Notes#

• The consistency of scikit-learn's fit/transform/predict API is what makes pipelines, cross-validation, and hyperparameter search all work together seamlessly — understanding this API is understanding why scikit-learn is organized the way it is.
• Custom transformers must never learn from data inside transform() — all learning happens in fit(), or cross-validation results become invalid due to data leakage.
• Always save (and load) the entire pipeline, not just the final estimator — this guarantees new data is preprocessed identically to how training data was, every time.
• Recording model metadata (library versions, hyperparameters, training date) alongside a saved model is essential for long-term maintainability, not just a nice-to-have.
• This entire workflow — from split, through preprocessing, comparison, tuning, and final evaluation — mirrors the methodology introduced in the very first notebook of this course; every subsequent notebook has been building toward being able to execute it correctly, end to end.

Best Practices Checklist#

• Split data into train/test before any preprocessing step is fit
• Build all preprocessing as a ColumnTransformer inside a Pipeline, never as separate manual steps
• Write custom transformers following the fit/transform convention when built-in tools don't cover a specific need
• Compare multiple model types via cross-validation before committing to hyperparameter tuning
• Choose the evaluation metric based on the problem's actual characteristics (class balance, cost asymmetry), not by default
• Touch the test set exactly once, after all model selection and tuning is complete
• Save the entire pipeline (preprocessing + model) with joblib, not just the final estimator
• Record metadata (versions, hyperparameters, training date) alongside every saved model
• Use get_feature_names_out() to correctly match feature importance or coefficients back to real feature names after encoding