Notebook 2: Data Preprocessing and Feature Engineering

Module: Machine Learning Level: Beginner to Intermediate Prerequisites: Notebook 1 (Statistical Learning Foundations) Estimated Time: 3–4 hours

---

Table of Contents#

1. Why Preprocessing Matters
2. The Golden Rule: Fit on Train, Transform Everywhere
3. Handling Missing Data
4. Feature Scaling and Normalization
5. Encoding Categorical Variables
6. Feature Engineering Techniques
7. Feature Selection
8. Hands-On: Building a Preprocessing Pipeline
9. Case Studies
10. Common Misconceptions and Pitfalls
11. Glossary
12. Exercises
13. Quiz
14. Summary
15. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Explain why data leakage during preprocessing is one of the most common causes of inflated performance estimates
• Choose an appropriate strategy for handling missing data, and understand the assumptions behind each
• Apply the correct scaling method (standardization vs. min-max normalization) for a given algorithm
• Encode categorical variables correctly, avoiding both information loss and artificial ordering
• Engineer new features from raw data using domain reasoning, interaction terms, and transformations
• Apply filter, wrapper, and embedded feature selection methods and explain their tradeoffs
• Build a leakage-safe preprocessing pipeline using scikit-learn's Pipeline and ColumnTransformer

---

1. Why Preprocessing Matters#

Real-world data is almost never in a form a model can consume directly. It arrives with missing values, features on wildly different scales, categorical labels instead of numbers, and often contains far more noise than signal. It is a widely repeated observation among practitioners that the majority of time in an applied machine learning project is spent on data preparation rather than model selection — and the quality of that preparation frequently has a larger effect on final performance than the choice of algorithm.

This notebook covers the practical skills that sit between "raw data" and "data a model in Notebooks 3 onward can actually learn from." Crucially, it also introduces the single most consequential rule in applied preprocessing: doing these steps in the wrong order silently reintroduces the exact data leakage problem discussed in Notebook 1, Section 10.

---

2. The Golden Rule: Fit on Train, Transform Everywhere#

2.1 The Rule#

Every preprocessing step that learns something from data (a mean and standard deviation for scaling, a set of categories for encoding, a median for imputation) must learn those values only from the training set, and then apply the exact same learned values to the validation and test sets.

$\text{Fit: } \text{transform}_\theta \leftarrow \text{learn from } X_{\text{train}} \text{ only}$ $\text{Apply: } X_{\text{train}}', X_{\text{val}}', X_{\text{test}}' \leftarrow \text{transform}_\theta(X_{\text{train}}), \text{transform}_\theta(X_{\text{val}}), \text{transform}_\theta(X_{\text{test}})$

2.2 Why This Matters#

If you compute, for example, the mean and standard deviation for standardization using the entire dataset (train plus validation plus test combined), then information about the validation and test distributions has leaked into the numbers used to transform the training set. The model's validation and test performance will then be a slightly-to-significantly optimistic estimate of true generalization performance, exactly the problem introduced in Notebook 1, Section 10.

2.3 A Minimal Illustration of the Mistake#

Python
CopyAsk AI
# INCORRECT: leakage-prone approach
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # fit on ALL data, including future test rows

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
# By this point, X_test has already influenced the scaler's mean and std.

Python
CopyAsk AI
# CORRECT: leakage-safe approach
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit ONLY on training data
X_test_scaled = scaler.transform(X_test)          # apply the same transform, no new fitting

Section 8 shows how scikit-learn's Pipeline enforces this rule automatically, which is the recommended approach for any real project.

---

3. Handling Missing Data#

3.1 Why Data Goes Missing#

Understanding why data is missing helps you choose the right strategy. Statisticians typically classify missingness into three types:

• Missing Completely at Random (MCAR): the fact that a value is missing has nothing to do with any variable, observed or unobserved (e.g., a sensor randomly drops a reading due to a transient network glitch).
• Missing at Random (MAR): the probability of missingness depends on other observed variables, but not on the missing value itself (e.g., older survey respondents are less likely to answer an income question, but conditional on age, missingness is unrelated to actual income).
• Missing Not at Random (MNAR): the probability of missingness depends on the missing value itself (e.g., people with very high income are less likely to disclose it).

Most standard imputation techniques implicitly assume MCAR or MAR. Under MNAR, naive imputation can introduce systematic bias, and this possibility is worth flagging explicitly to domain experts rather than silently imputing.

3.2 Strategies for Handling Missing Data#

3.3 Numeric Example#

Python
CopyAsk AI
import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer

df = pd.DataFrame({
    "age": [25, np.nan, 35, 40, np.nan, 29],
    "income": [50000, 62000, np.nan, 81000, 45000, 58000]
})

imputer = SimpleImputer(strategy="median")
df_imputed = pd.DataFrame(
    imputer.fit_transform(df), columns=df.columns
)
print(df_imputed)

Expected output:

code
Ask AICopy
    age   income
0  25.0  50000.0
1  32.0  62000.0
2  35.0  59166.7
3  40.0  81000.0
4  32.0  45000.0
5  29.0  58000.0

Notice that the median used to fill each column is computed from that column's observed values, which is why age fills with 32 (the median of 25, 35, 40, 29) and income fills with roughly 59,166.7 (the median of the observed income values).

3.4 Preserving the "Missingness Signal"#

Sometimes the fact that a value is missing is itself predictive. For example, in a loan application, an applicant declining to report their income may itself correlate with default risk. Adding a binary indicator column (income_was_missing) alongside an imputed value lets the model use both the imputed estimate and the fact of missingness as separate signals.

Python
CopyAsk AI
df["income_was_missing"] = df["income"].isna().astype(int)

---

4. Feature Scaling and Normalization#

4.1 Why Scale at All#

Many algorithms are sensitive to the scale of input features:

• Distance-based algorithms (k-Nearest Neighbors, Notebook 8; K-Means, Notebook 9) compute distances between points. A feature measured in the tens of thousands (e.g., income) will dominate a feature measured in single digits (e.g., number of children) purely due to scale, not actual importance.
• Gradient-based optimization (Logistic Regression, Notebook 4; Support Vector Machines, Notebook 7) converges faster and more reliably when features are on comparable scales.
• Regularized models (Ridge, Lasso, Notebook 5) apply penalties to coefficient magnitudes; if features are on different scales, the penalty affects them unevenly regardless of true importance.

Tree-based models (Decision Trees, Random Forests, Gradient Boosting, Notebook 6) are a notable exception: splits are based on threshold comparisons within a single feature at a time, so scaling has no effect on their performance.

4.2 Standardization (Z-score Normalization)#

Transforms a feature to have mean 0 and standard deviation 1:

$x' = \frac{x - \mu}{\sigma}$

where $\mu$ and $\sigma$ are the mean and standard deviation, computed on the training set only (Section 2).

Python
CopyAsk AI
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Use standardization when the algorithm assumes roughly normally distributed features, or in general as a solid default for linear models, SVMs, and neural networks.

4.3 Min-Max Normalization#

Rescales a feature to a fixed range, typically [0, 1]:

$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$

Python
CopyAsk AI
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Use min-max normalization when you need bounded values (e.g., for certain neural network activation functions) or when the data does not follow a roughly normal distribution. It is more sensitive to outliers than standardization, since a single extreme value stretches the entire range.

4.4 Robust Scaling#

Uses the median and interquartile range (IQR) instead of the mean and standard deviation, making it far less sensitive to outliers:

$x' = \frac{x - \text{median}}{\text{IQR}}$

Python
CopyAsk AI
from sklearn.preprocessing import RobustScaler

scaler = RobustScaler()
X_train_scaled = scaler.fit_transform(X_train)

Use robust scaling when your data contains significant outliers that you do not want to remove but also do not want dominating the scale computation.

4.5 Comparison Table#

---

5. Encoding Categorical Variables#

5.1 Why Encoding Is Necessary#

Most ML algorithms require numeric input. Categorical features (e.g., "color": red, green, blue) must be converted to numbers, but naively assigning arbitrary integers (red=1, green=2, blue=3) introduces a false sense of order and magnitude that the algorithm may misinterpret (e.g., treating blue as "three times" red).

5.2 One-Hot Encoding#

Creates a new binary column for each category:

Python
CopyAsk AI
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
encoded = encoder.fit_transform(X_train[["color"]])

Use one-hot encoding for nominal categories (no inherent order), such as color, country, or product category. The main downside is dimensionality: a feature with hundreds of unique categories produces hundreds of new columns, which can be problematic for some algorithms (this is sometimes called the "curse of dimensionality," touched on further in Notebook 10).

The handle_unknown="ignore" parameter is important in practice: it ensures that if the test or production data contains a category never seen during training, the encoder does not raise an error, instead encoding it as all zeros.

5.3 Ordinal Encoding#

Assigns integers that respect a genuine, meaningful order:

Python
CopyAsk AI
from sklearn.preprocessing import OrdinalEncoder

# Explicitly specify the true order
encoder = OrdinalEncoder(categories=[["low", "medium", "high"]])
encoded = encoder.fit_transform(X_train[["risk_level"]])

Use ordinal encoding only when categories have a genuine rank (e.g., education level: high school < bachelor's < master's < doctorate; survey responses: strongly disagree < disagree < neutral < agree < strongly agree). Applying ordinal encoding to a nominal feature (like color) would incorrectly imply an order that does not exist.

5.4 Target Encoding (Mean Encoding)#

Replaces each category with the mean of the target variable for that category, computed only on the training set:

Python
CopyAsk AI
# Simplified illustration (production use typically requires
# cross-validated / smoothed target encoding to avoid leakage)
train_means = X_train.groupby("city")["y"].mean()
X_train["city_encoded"] = X_train["city"].map(train_means)
X_test["city_encoded"] = X_test["city"].map(train_means)

Target encoding is powerful for high-cardinality categorical features (e.g., zip code, with thousands of unique values), where one-hot encoding would be impractical. However, it is also one of the most common sources of subtle data leakage in real projects: computing the mean naively on the full training set before cross-validation folds are created leaks target information within the training process itself. Production-grade implementations use out-of-fold encoding or additive smoothing toward the global mean to mitigate this.

5.5 Comparison Table#

---

6. Feature Engineering Techniques#

Feature engineering is the process of creating new input features from raw data to make patterns more accessible to a model. This is frequently where domain expertise adds the most value in a machine learning project.

6.1 Mathematical Transformations#

• Log transform: $x' = \log(x + 1)$, useful for right-skewed distributions (e.g., income, transaction amounts), compressing large values and making the distribution more symmetric.
• Polynomial features: creating $x^2$, $x^3$, or interaction terms $x_1 \cdot x_2$ to let a linear model capture nonlinear relationships (used directly in the polynomial regression example in Notebook 1, Section 7).
• Binning/discretization: converting a continuous feature into categorical buckets (e.g., age into "18-25," "26-40," "41-65," "65+"), which can help simple models capture non-monotonic relationships.

6.2 Interaction Features#

Sometimes the combination of two features carries more signal than either alone. For example, "price per square foot" (price divided by square footage) can be more predictive for a housing model than price and square footage as separate features, because it directly captures a meaningful ratio a linear model would otherwise have to approximate indirectly.

Python
CopyAsk AI
df["price_per_sqft"] = df["price"] / df["sqft"]

6.3 Date and Time Features#

Raw timestamps are rarely useful directly, but decomposed components often are:

Python
CopyAsk AI
df["hour_of_day"] = df["timestamp"].dt.hour
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
df["month"] = df["timestamp"].dt.month

These derived features let models capture cyclical patterns (e.g., weekend versus weekday shopping behavior) that a raw timestamp cannot express directly.

6.4 Domain-Specific Feature Engineering#

The most impactful feature engineering usually comes from understanding the problem domain rather than generic transformations. In the predictive maintenance case study from Notebook 0, raw vibration sensor readings were transformed into rolling averages and rates of change — features that directly encode the physical concept of "the machine is degrading," which the raw signal alone does not make explicit to a model.

---

7. Feature Selection#

7.1 Why Select Features#

Including irrelevant or redundant features can hurt model performance (adding noise, increasing variance per the bias-variance tradeoff from Notebook 1), increase training time, and make models harder to interpret. Feature selection aims to keep only the features that meaningfully contribute to prediction.

7.2 Filter Methods#

Select features based on a statistical property, independent of any specific model:

• Correlation with target: keep features with high absolute correlation with the target variable
• Variance threshold: remove features with very low variance (they carry little information, since they barely change across examples)
• Chi-squared test: for categorical features and classification targets, measures statistical dependence

Python
CopyAsk AI
from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=0.01)
X_train_selected = selector.fit_transform(X_train)

Filter methods are fast and model-agnostic, but they evaluate each feature somewhat in isolation and can miss useful interaction effects between features.

7.3 Wrapper Methods#

Evaluate subsets of features by actually training a model on each subset and measuring performance:

• Recursive Feature Elimination (RFE): repeatedly trains a model, removes the least important feature, and repeats until the desired number of features remains

Python
CopyAsk AI
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=5000)
selector = RFE(model, n_features_to_select=5)
selector.fit(X_train, y_train)
print(f"Selected features mask: {selector.support_}")

Wrapper methods tend to find better-performing feature subsets than filter methods because they account for how features interact with a specific model, but are considerably more computationally expensive since they require training many models.

7.4 Embedded Methods#

Feature selection happens as an intrinsic part of model training:

• Lasso regression (Notebook 5) drives some coefficients to exactly zero as part of its regularization penalty, effectively performing feature selection during training.
• Tree-based feature importance (Notebook 6): decision trees and ensembles naturally rank features by how much they reduce impurity across splits, which can be used to filter out low-importance features after training.

Python
CopyAsk AI
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

importances = model.feature_importances_
print(f"Feature importances: {importances}")

Embedded methods offer a good balance of computational efficiency and model-awareness, and are often the practical default in industry pipelines.

7.5 Comparison Table#

---

8. Hands-On: Building a Preprocessing Pipeline#

This section combines everything above into a single, leakage-safe scikit-learn pipeline using ColumnTransformer, which is the recommended pattern for any real project.

Python
CopyAsk AI
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

# Sample raw data with a mix of numeric, categorical, and missing values
data = pd.DataFrame({
    "age": [25, 45, np.nan, 35, 52, 23, 41, np.nan],
    "income": [50000, 82000, 61000, np.nan, 95000, 43000, 71000, 58000],
    "city": ["NY", "LA", "NY", "SF", "LA", "SF", "NY", "LA"],
    "purchased": [0, 1, 0, 1, 1, 0, 1, 0]
})

X = data.drop(columns=["purchased"])
y = data["purchased"]

numeric_features = ["age", "income"]
categorical_features = ["city"]

# Build separate pipelines for numeric and categorical columns
numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ("onehot", OneHotEncoder(handle_unknown="ignore"))
])

# Combine them, routing each column to the right pipeline
preprocessor = ColumnTransformer(transformers=[
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
])

# Chain preprocessing and the model into one pipeline
full_pipeline = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=5000))
])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# A single .fit() call correctly fits every imputer, scaler, and encoder
# ONLY on X_train, then applies the learned transforms to X_test
# automatically, with zero risk of accidentally leaking test data.
full_pipeline.fit(X_train, y_train)

print(f"Test accuracy: {full_pipeline.score(X_test, y_test):.2%}")

Why this pattern matters: Calling full_pipeline.fit(X_train, y_train) fits the imputer's median, the scaler's mean and standard deviation, and the encoder's category list all on X_train alone. Calling full_pipeline.score(X_test, y_test) (or .predict(X_test)) automatically applies those already-fitted transforms to the test data without refitting them. This single pattern structurally enforces the golden rule from Section 2, making leakage far harder to introduce by accident. This same pipeline pattern will be reused, extended with hyperparameter tuning, in Notebook 12.

---

9. Case Studies#

9.1 Case Study: A Housing Price Model Inflated by Leakage#

A team building a house price prediction model scales all numeric features (including square footage and price-related features) using StandardScaler fit on the entire dataset, then splits into train and test. Their reported test R-squared is unusually high compared to published benchmarks on similar data.

Diagnosis: Because the scaler was fit on the combined dataset before splitting (violating Section 2), the mean and standard deviation used to scale the training data already encode information about the test set's distribution. This is a mild form of leakage that, in this instance, was compounded by an even larger error: a "price_per_sqft" engineered feature (Section 6.2) was computed using the target variable itself, meaning the model had indirect access to the answer during training.

Lesson: Any feature engineered from the target variable, however indirectly, must be recomputed carefully to avoid leaking the target into the inputs, and all fitted transformations must be scoped to the training set only, exactly as shown in Section 8's pipeline.

9.2 Case Study: Target Encoding Leakage in a Kaggle Competition#

Multiple public post-mortems from data science competitions describe teams achieving excellent cross-validation scores using naive target encoding (Section 5.4), only to see performance collapse on the actual held-out leaderboard data.

Diagnosis: When target encoding is computed on the full training set before cross-validation folds are created, each fold's "training" portion contains encoded values that were partly derived from that same fold's target values, creating leakage within the cross-validation procedure itself, not just between train and test.

Lesson: Target encoding requires special care beyond the general fit-on-train rule: it should be computed using out-of-fold statistics (encoding each fold using only the other folds' data) or heavily smoothed toward the global mean, particularly for categories with few observations.

9.3 Case Study: Missing Data as a Signal in Credit Applications#

A bank building a credit default model initially drops all rows with missing "years_at_current_job" values, assuming the data is Missing Completely at Random. Model performance improves substantially after instead imputing the missing values and adding a "years_at_current_job_was_missing" indicator feature (Section 3.4).

Diagnosis: Applicants who leave this field blank turn out to disproportionately include gig workers and recently unemployed individuals — meaning the missingness itself, per Section 3.1, is Missing Not at Random and directly informative about default risk. Deleting these rows both discarded useful signal and reduced the training set size.

Lesson: Always investigate why data might be missing (Section 3.1) before defaulting to deletion, since the pattern of missingness can itself be one of the more predictive features available.

---

10. Common Misconceptions and Pitfalls#

---

11. Glossary#

---

12. Exercises#

Exercise 1 (Conceptual)#

A dataset has a "survey_response_time_missing" pattern where younger respondents are far more likely to skip a particular question than older respondents, but among people of the same age, the chance of skipping the question is unrelated to their actual answer. Classify this missingness type and justify your answer.

<details> <summary>Solution</summary>

This is Missing at Random (MAR): the probability of missingness depends on an observed variable (age), but conditional on age, it does not depend on the value that would have been observed. This differs from MNAR, where the missingness would depend on the (unobserved) answer itself, for example if people with extreme opinions were less likely to answer regardless of age.

</details>

---

Exercise 2 (Conceptual)#

Explain why applying OneHotEncoder to a "satisfaction_level" feature with categories "low," "medium," "high" might be a worse choice than OrdinalEncoder for a linear regression model.

<details> <summary>Solution</summary>

"Satisfaction level" has a genuine, meaningful order (low < medium < high), per Section 5.3. One-hot encoding discards this order entirely, treating the three categories as unrelated, which forces a linear model to learn three independent, unconstrained coefficients and cannot express the natural monotonic relationship (as satisfaction increases, some outcome tends to increase or decrease smoothly). Ordinal encoding, by preserving the order as 0, 1, 2, allows a linear model to directly capture that monotonic trend with a single coefficient.

</details>

---

Exercise 3 (Coding)#

Given a ColumnTransformer similar to the one in Section 8, add a VarianceThreshold filter step (Section 7.2) to the numeric pipeline that removes numeric features with variance below 0.02. Where in the pipeline should this step go relative to scaling, and why?

<details> <summary>Solution</summary>

Python
CopyAsk AI
from sklearn.feature_selection import VarianceThreshold

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("variance_filter", VarianceThreshold(threshold=0.02))
])

The variance filter should generally go after scaling, not before. Variance is scale-dependent: a feature measured in the thousands could have huge raw variance despite being nearly constant in relative terms, while a feature measured in small decimals could have tiny raw variance despite carrying real signal. Filtering on variance before scaling can accidentally keep or discard features based on their original units rather than their actual informativeness. Note that after standardization, every feature has variance close to 1 by construction, so a variance filter applied strictly after StandardScaler would remove almost nothing; in practice, variance filtering is more commonly applied before scaling but after ensuring features are in comparable units, or as a filter based on domain-specific variance thresholds appropriate to each feature's original scale.

</details>

---

Exercise 4 (Applied Thinking)#

You are building a churn prediction model and consider adding a "customer_lifetime_value" feature, which is computed using the customer's total historical revenue up to and including the prediction date. Explain a scenario in which this feature could leak information about the churn label, and how you would fix it.

<details> <summary>Solution</summary>

If "customer_lifetime_value" is computed using data up to the current date, including revenue recorded after the point in time you are trying to predict from, it could implicitly encode the outcome. For example, if churned customers stop generating revenue, a lifetime value computed too close to or after the churn event would show a sudden drop that is a direct symptom of already having churned, not a genuine predictive signal available before the fact. The fix is to compute lifetime value strictly using data available as of a clearly defined prediction cutoff date, with a buffer, ensuring the feature reflects only information genuinely available at prediction time, not information from the future relative to that cutoff. This is a specific instance of a broader principle: any feature derived from data that could only exist after the event you are predicting must be excluded or carefully time-bounded.

</details>

---

13. Quiz#

Q1. Which of the following correctly follows the golden rule from Section 2? a) Fit a StandardScaler on the full dataset, then split into train and test b) Fit a StandardScaler on the training set, then use it to transform both training and test sets c) Fit separate StandardScaler instances on the training set and the test set independently d) Skip scaling entirely if using logistic regression

Q2. Which type of missing data mechanism means the probability of missingness depends on the missing value itself? a) MCAR b) MAR c) MNAR d) None of the above

Q3. Which encoding method is most appropriate for a "shirt_size" feature with categories "small," "medium," "large"? a) One-hot encoding b) Ordinal encoding c) Target encoding d) No encoding is needed

Q4. Which feature selection approach evaluates the actual performance of a model trained on different feature subsets? a) Filter method b) Wrapper method c) Embedded method d) Variance threshold

Q5. True or False: Tree-based models like Random Forests generally require feature scaling to perform well.

<details> <summary>Answers</summary>

1. b) Fit a StandardScaler on the training set, then use it to transform both training and test sets
2. c) MNAR (Missing Not at Random)
3. b) Ordinal encoding, since shirt sizes have a genuine order
4. b) Wrapper method
5. False — tree-based models split on threshold comparisons within a single feature and are unaffected by monotonic scaling, per Section 4.1

</details>

---

14. Summary#

• The golden rule of preprocessing is to fit every transformation only on the training data, then apply the same fitted transformation to validation and test data, to avoid reintroducing data leakage.
• Missing data should be diagnosed (MCAR, MAR, or MNAR) before choosing a handling strategy, since the pattern of missingness can itself be predictive.
• Feature scaling matters for distance-based, gradient-based, and regularized models, but not for tree-based models.
• Categorical encoding method should match the nature of the category: one-hot for nominal, ordinal for ordered, and target encoding (with care) for high-cardinality features.
• Feature engineering, including transformations, interaction terms, and domain-specific derived features, is often where the largest performance gains in a real project come from.
• Feature selection methods (filter, wrapper, embedded) trade off speed against the ability to capture feature interactions and model-specific importance.
• scikit-learn's Pipeline and ColumnTransformer structurally enforce leakage-safe preprocessing and should be the default pattern for any real project.

---

15. Further Reading#

• scikit-learn documentation on preprocessing: https://scikit-learn.org/stable/modules/preprocessing.html
• scikit-learn documentation on ColumnTransformer and Pipeline: https://scikit-learn.org/stable/modules/compose.html
• Max Kuhn and Kjell Johnson, Feature Engineering and Selection: A Practical Approach for Predictive Models
• Roderick J. A. Little and Donald B. Rubin, Statistical Analysis with Missing Data — the foundational reference for MCAR/MAR/MNAR

---

Next Up#

Notebook 3: Linear Regression — simple and multiple regression, the assumptions behind Ordinary Least Squares, and an introduction to regularization (Ridge and Lasso), building directly on the empirical risk minimization framework from Notebook 0 and the preprocessing pipeline from this notebook.

---

Course: Machine Learning | Notebook 2 of 13