Data Cleaning & Preprocessing Best Practices

1. Overview#

Real-world data is noisy, incomplete, and inconsistent. High-performing machine learning models require rigorous preprocessing: handling missingness (MCAR, MAR, MNAR), outlier mitigation, and variance stabilization.

Never fit data imputers or scalers on the entire dataset. Always fit on the training split and transform the validation/test splits to prevent Data Leakage.

---

2. Missing Value Imputation Strategies#

---

3. Outlier Detection with Interquartile Range (IQR) & Z-Score#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import pandas as pd
import numpy as np

def cap_outliers_iqr(df: pd.DataFrame, column: str, multiplier: float = 1.5) -> pd.Series:
 """Winsorizes numerical values outside the IQR bounds."""
 Q1 = df[column].quantile(0.25)
 Q3 = df[column].quantile(0.75)
 IQR = Q3 - Q1
 lower_bound = Q1 - multiplier * IQR
 upper_bound = Q3 + multiplier * IQR
 
 return df[column].clip(lower=lower_bound, upper=upper_bound)

---

4. Best Practices Checklist#

• Inspect missingness patterns with df.isnull().sum() and missingno matrix visualizations.
• Standardize string columns (strip whitespace, lowercase, normalize unicode accents).
• Validate column data types before persisting to Parquet or feeding into ML pipelines.