Model Evaluation and Metrics

Overview#

A model is only as trustworthy as the metric used to evaluate it — and picking the wrong metric can make a genuinely bad model look great, or vice versa. This notebook covers classification metrics (accuracy, precision, recall, F1, the confusion matrix, ROC-AUC), regression metrics (MSE, RMSE, MAE, R²), and — critically — guidance on which metric fits which situation, since this is where many real-world evaluation mistakes happen.

1. Why Accuracy Alone Can Be Deeply Misleading#

Python
CopyAsk AI
import numpy as np
from sklearn.metrics import accuracy_score

# Simulating a highly imbalanced problem: 950 negative, 50 positive
y_true = np.array([0] * 950 + [1] * 50)

# A "model" that just predicts the majority class every single time
y_pred_lazy = np.array([0] * 1000)

print(f"Accuracy: {accuracy_score(y_true, y_pred_lazy):.3f}")   # 0.950 !!

A model that never even looks at the input and always predicts "0" achieves 95% accuracy on this data — clearly not a useful model, but accuracy alone makes it look excellent. This is the single most important lesson in model evaluation: accuracy is only meaningful when classes are roughly balanced; for imbalanced problems (fraud detection, disease diagnosis, rare event prediction), it can be actively deceptive.

2. The Confusion Matrix: The Foundation of Classification Metrics#

Python
CopyAsk AI
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

y_true = [0, 1, 1, 0, 1, 1, 0, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1, 1, 0, 1, 0]

cm = confusion_matrix(y_true, y_pred)
print(cm)

ConfusionMatrixDisplay(confusion_matrix=cm).plot()
plt.show()

text
Ask AICopy
                Predicted 0    Predicted 1
Actual 0            TN              FP
Actual 1            FN              TP

• True Positive (TP): correctly predicted positive
• True Negative (TN): correctly predicted negative
• False Positive (FP): predicted positive, actually negative (a "false alarm" — Type I error)
• False Negative (FN): predicted negative, actually positive (a "missed detection" — Type II error)

Every classification metric covered below is derived directly from these four numbers — understanding the confusion matrix is the foundation for understanding everything that follows.

3. Precision and Recall#

Python
CopyAsk AI
from sklearn.metrics import precision_score, recall_score

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")

Mathematical Formulation
Explain FormulaCopy
Precision = TP / (TP + FP)
  "Of everything I predicted positive, how much was actually positive?"
  High precision = few false alarms

Recall = TP / (TP + FN)
  "Of everything that was actually positive, how much did I catch?"
  High recall = few missed detections

The precision-recall tradeoff#

Python
CopyAsk AI
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, weights=[0.9, 0.1], random_state=42)
model = LogisticRegression().fit(X, y)

y_proba = model.predict_proba(X)[:, 1]   # predicted probability of class 1

for threshold in [0.2, 0.5, 0.8]:
    y_pred_at_threshold = (y_proba >= threshold).astype(int)
    p = precision_score(y, y_pred_at_threshold)
    r = recall_score(y, y_pred_at_threshold)
    print(f"Threshold={threshold}: precision={p:.3f}, recall={r:.3f}")

text
Ask AICopy
Threshold=0.2: precision=0.412, recall=0.891   <- lower threshold: catches more positives, more false alarms too
Threshold=0.5: precision=0.667, recall=0.723   <- balanced default
Threshold=0.8: precision=0.895, recall=0.451   <- higher threshold: fewer false alarms, misses more positives

Precision and recall move in opposite directions as the classification threshold changes — there is no threshold that maximizes both simultaneously (except in an unrealistically perfect model). Which one to prioritize depends entirely on the real-world cost of each error type.

When to prioritize precision vs recall#

text
Ask AICopy
Prioritize RECALL when missing a positive is costly:
- Cancer/disease screening (missing a sick patient is far worse than a
  false alarm that leads to more testing)
- Fraud detection (missing actual fraud costs money directly)

Prioritize PRECISION when false alarms are costly:
- Spam filtering (marking a legitimate important email as spam is very
  disruptive to the user)
- Content moderation flagging accounts for permanent bans (wrongly
  banning a legitimate user is a serious cost)

4. F1 Score: Balancing Precision and Recall#

Python
CopyAsk AI
from sklearn.metrics import f1_score

f1 = f1_score(y_true, y_pred)
print(f"F1 score: {f1:.3f}")

Mathematical Formulation
Explain FormulaCopy
F1 = 2 * (precision * recall) / (precision + recall)

F1 is the harmonic mean of precision and recall — it penalizes extreme imbalance between the two more heavily than a simple average would (a model with precision=1.0 and recall=0.01 gets an F1 near 0.02, not 0.505), making it a useful single-number summary when both matter somewhat, and neither error type is dramatically more costly than the other.

F-beta: weighting precision and recall unevenly#

Python
CopyAsk AI
from sklearn.metrics import fbeta_score

f2 = fbeta_score(y_true, y_pred, beta=2)     # weights recall higher than precision
f0_5 = fbeta_score(y_true, y_pred, beta=0.5)   # weights precision higher than recall

beta > 1 weights recall more heavily; beta < 1 weights precision more heavily — useful when you know the relative cost imbalance but still want a single combined metric rather than tracking precision and recall separately.

5. The Full Classification Report#

Python
CopyAsk AI
from sklearn.metrics import classification_report

print(classification_report(y_true, y_pred))

Mathematical Formulation
Explain FormulaCopy
              precision    recall  f1-score   support

           0       0.75      0.75      0.75         4
           1       0.83      0.83      0.83         6

    accuracy                           0.80        10
   macro avg       0.79      0.79      0.79        10
weighted avg       0.80      0.80      0.80        10

macro avg averages the metric equally across classes (treats each class as equally important, regardless of size); weighted avg averages proportional to each class's frequency (support) — macro avg is generally more informative for imbalanced classes since it doesn't let the majority class dominate the reported average.

6. ROC Curves and AUC#

The ROC (Receiver Operating Characteristic) curve plots the True Positive Rate (recall) against the False Positive Rate at every possible classification threshold, visualizing the full tradeoff rather than a single point.

Python
CopyAsk AI
from sklearn.metrics import roc_curve, roc_auc_score, RocCurveDisplay

fpr, tpr, thresholds = roc_curve(y, y_proba)
auc = roc_auc_score(y, y_proba)

RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=auc).plot()
plt.plot([0, 1], [0, 1], linestyle="--", label="Random guessing")
plt.legend()
plt.show()

print(f"AUC: {auc:.3f}")

text
Ask AICopy
AUC (Area Under the Curve) interpretation:
  1.0  → perfect classifier
  0.5  → no better than random guessing (the diagonal line)
  0.0  → perfectly WRONG (perfectly anti-correlated — flip predictions
          to get a perfect classifier)

AUC can be interpreted as: the probability that the model ranks a
RANDOMLY CHOSEN positive example higher than a randomly chosen negative
example.

AUC is threshold-independent — it summarizes performance across every possible threshold at once, which makes it useful for comparing models generally, but less directly actionable than precision/recall at a specific, chosen operating threshold.

ROC-AUC's limitation with imbalanced data#

Python
CopyAsk AI
from sklearn.metrics import precision_recall_curve, average_precision_score

precision_vals, recall_vals, _ = precision_recall_curve(y, y_proba)
avg_precision = average_precision_score(y, y_proba)
print(f"Average Precision (PR-AUC): {avg_precision:.3f}")

For heavily imbalanced datasets, ROC-AUC can look deceptively good even for a mediocre model, because the true negative rate dominates the false positive rate calculation when negatives vastly outnumber positives — the Precision-Recall curve (and its AUC) is generally the more informative choice for imbalanced classification problems.

7. Multiclass Classification Metrics#

Python
CopyAsk AI
from sklearn.metrics import classification_report
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)   # 3 classes
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
y_pred = model.predict(X_test)

print(classification_report(y_test, y_pred, target_names=load_iris().target_names))

For multiclass problems, precision/recall/F1 are computed per-class (treating each class as "this class vs everything else" in turn), then optionally averaged using macro, weighted, or micro strategies — micro averaging pools all TP/FP/FN across classes before computing the metric, which is equivalent to overall accuracy in the single-label multiclass case.

8. Regression Metrics#

Python
CopyAsk AI
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np

y_true_reg = np.array([3.0, -0.5, 2.0, 7.0])
y_pred_reg = np.array([2.5, 0.0, 2.1, 7.8])

mse = mean_squared_error(y_true_reg, y_pred_reg)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_true_reg, y_pred_reg)
r2 = r2_score(y_true_reg, y_pred_reg)

print(f"MSE: {mse:.3f}")
print(f"RMSE: {rmse:.3f}")
print(f"MAE: {mae:.3f}")
print(f"R²: {r2:.3f}")

text
Ask AICopy
MSE (Mean Squared Error): average of squared errors
  - Penalizes large errors much more heavily than small ones (squaring)
  - Not in the same units as the target (units squared)

RMSE (Root Mean Squared Error): square root of MSE
  - Back in the same units as the target — more interpretable than MSE
  - Still penalizes large errors more heavily than MAE does

MAE (Mean Absolute Error): average of absolute errors
  - Same units as the target, treats all error sizes proportionally
  - More robust to outliers than MSE/RMSE, since it doesn't square errors

R² (coefficient of determination): proportion of variance in the target
  explained by the model
  - 1.0 = perfect predictions
  - 0.0 = no better than always predicting the mean
  - Can go NEGATIVE for a model that's worse than just predicting the mean

Choosing between MSE/RMSE and MAE#

Python
CopyAsk AI
# If large errors are disproportionately costly (e.g. predicting demand
# for inventory — a huge under-prediction is much worse than several
# small ones), MSE/RMSE's heavier penalty on large errors is appropriate.

# If all errors should be weighted proportionally, and outliers shouldn't
# dominate the metric, MAE is more appropriate.

Adjusted R²#

Python
CopyAsk AI
def adjusted_r2(r2, n_samples, n_features):
    return 1 - (1 - r2) * (n_samples - 1) / (n_samples - n_features - 1)

adj_r2 = adjusted_r2(r2_score(y_true_reg, y_pred_reg), n_samples=4, n_features=1)
print(f"Adjusted R²: {adj_r2:.3f}")

Plain R² always increases (or stays the same) as more features are added to a model, even entirely useless ones — adjusted R² penalizes adding features that don't meaningfully improve fit, making it more appropriate than plain R² for comparing models with different numbers of features.

9. Choosing the Right Metric: A Decision Guide#

Architecture & Data Flow
Copy
CLASSIFICATION:
├── Classes roughly balanced, all errors equally costly?
│   └── Accuracy is a reasonable primary metric
├── Classes imbalanced, or errors have different costs?
│   ├── Missing positives is worse (disease screening, fraud) → prioritize RECALL
│   ├── False alarms are worse (spam, content moderation) → prioritize PRECISION
│   └── Both matter, roughly balanced → F1 score
├── Need to compare models across ALL thresholds?
│   ├── Classes roughly balanced → ROC-AUC
│   └── Classes imbalanced → Precision-Recall AUC (average precision)

REGRESSION:
├── Large errors should be penalized heavily → RMSE
├── All errors should count proportionally, outlier-robust → MAE
├── Need a normalized, interpretable "how good is this" measure → R²
└── Comparing models with different numbers of features → Adjusted R²

Notes#

• Accuracy is dangerously misleading on imbalanced data — always check class balance before choosing accuracy as a primary metric.
• Precision and recall trade off against each other via the classification threshold — there's no free lunch; the right balance depends on the real-world cost of each error type, a business/domain decision as much as a technical one.
• ROC-AUC summarizes performance across every threshold but can look deceptively good on imbalanced data; Precision-Recall AUC is usually more informative in that situation.
• Plain R² will never decrease when adding features, even useless ones — use adjusted R² when comparing models with different feature counts.
• No single metric captures everything — reporting multiple relevant metrics (e.g., a full classification report, not just one number) gives a much more honest picture of model performance than a single headline number.

Best Practices Checklist#

• Check class balance before relying on accuracy as a primary metric
• Report precision, recall, and F1 together, not just accuracy, for classification tasks
• Determine whether precision or recall matters more based on the real-world cost of false positives vs false negatives
• Use Precision-Recall AUC instead of ROC-AUC for heavily imbalanced classification problems
• Use classification_report with macro avg to get a fair picture across imbalanced classes
• Choose RMSE when large regression errors should be penalized more heavily; MAE when all errors should count proportionally
• Use adjusted R², not plain R², when comparing regression models with different numbers of features
• Report multiple relevant metrics rather than optimizing for a single headline number in isolation