Logistic Regression

Overview#

Despite the name, logistic regression is a classification algorithm, not a regression one — the name comes from its historical roots in regression and its close mathematical relationship to linear regression. This notebook goes deep into how logistic regression actually works: the sigmoid function and why it's used, how the model is fit via maximum likelihood (not squared error), decision boundaries, multiclass strategies, and a thorough look at its assumptions, strengths, and limitations.

1. From Linear Regression to Classification#

Linear regression predicts an unbounded real number. For classification, we need a prediction bounded between 0 and 1, interpretable as a probability. Logistic regression achieves this by passing a linear combination of features through the sigmoid (logistic) function.

Mathematical Formulation
Explain FormulaCopy
Step 1 (same as linear regression): z = w · x + b

Step 2 (the new part): p = sigmoid(z) = 1 / (1 + e^(-z))

Python
CopyAsk AI
import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z_values = np.linspace(-10, 10, 200)
plt.plot(z_values, sigmoid(z_values))
plt.axhline(y=0.5, color="r", linestyle="--")
plt.axvline(x=0, color="r", linestyle="--")
plt.xlabel("z (linear combination of features)")
plt.ylabel("sigmoid(z) — predicted probability")
plt.title("The Sigmoid Function")
plt.show()

text
Ask AICopy
Key properties of the sigmoid function:
- Output is always between 0 and 1 — valid as a probability
- sigmoid(0) = 0.5 — the "undecided" point
- As z → +∞, sigmoid(z) → 1
- As z → -∞, sigmoid(z) → 0
- It's an S-shaped ("sigmoid") curve, smooth and differentiable
  everywhere — essential for gradient-based optimization

Python
CopyAsk AI
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=2, n_redundant=0, n_clusters_per_class=1, random_state=42)

model = LogisticRegression()
model.fit(X, y)

probabilities = model.predict_proba(X[:5])
print(probabilities)   # each row: [P(class 0), P(class 1)]

2. The Decision Boundary#

Python
CopyAsk AI
predictions = model.predict(X[:5])
print(predictions)   # class predictions, derived by thresholding probability at 0.5

# Equivalent to:
manual_predictions = (model.predict_proba(X[:5])[:, 1] >= 0.5).astype(int)
print(manual_predictions)   # identical to model.predict()

The default decision boundary is where predicted probability equals 0.5 — which corresponds to exactly where z = w · x + b = 0 (since sigmoid(0) = 0.5). This means the decision boundary itself is linear — a straight line (or hyperplane in higher dimensions), even though the probability output is a smooth curve.

Python
CopyAsk AI
# Visualizing the linear decision boundary for 2D data
w1, w2 = model.coef_[0]
b = model.intercept_[0]

x1_range = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
x2_boundary = -(w1 * x1_range + b) / w2   # solving w1*x1 + w2*x2 + b = 0 for x2

plt.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", alpha=0.6)
plt.plot(x1_range, x2_boundary, "k-", linewidth=2, label="Decision boundary")
plt.legend()
plt.show()

This linear decision boundary is logistic regression's core limitation (and, in cases where it genuinely fits, its core strength for interpretability) — like linear regression, it can be extended to non-linear boundaries via polynomial/interaction feature engineering, but does not natively capture non-linear class separations the way trees or kernel SVMs do.

3. How the Model Is Fit: Maximum Likelihood Estimation#

Unlike linear regression's squared-error loss, logistic regression is fit by maximizing the likelihood of the observed labels given the model's predicted probabilities — equivalently, minimizing the log loss (also called binary cross-entropy).

text
Ask AICopy
Log loss for a single example:
  if y = 1:  loss = -log(p)          (p = predicted probability of class 1)
  if y = 0:  loss = -log(1 - p)

Combined into one formula:
  loss = -[y * log(p) + (1 - y) * log(1 - p)]

Python
CopyAsk AI
def log_loss_single(y_true, p_pred):
    epsilon = 1e-15   # avoid log(0)
    p_pred = np.clip(p_pred, epsilon, 1 - epsilon)
    return -(y_true * np.log(p_pred) + (1 - y_true) * np.log(1 - p_pred))

# Confident and correct: very low loss
print(log_loss_single(1, 0.99))   # ~0.01

# Confident and WRONG: very high loss
print(log_loss_single(1, 0.01))    # ~4.6

# Uncertain (p=0.5): moderate loss regardless of actual label
print(log_loss_single(1, 0.5))      # ~0.69

Why log loss instead of squared error for classification?#

text
Ask AICopy
1. Log loss heavily penalizes CONFIDENT WRONG predictions — a model that
   is 99% confident but wrong is punished far more severely than one
   that was uncertain and wrong. Squared error wouldn't distinguish these
   cases nearly as sharply.

2. Unlike OLS's squared-error loss, log loss combined with the sigmoid
   function does NOT have a closed-form solution — it must be minimized
   iteratively via gradient descent (or related optimization methods),
   because the resulting loss surface, while still convex, has no simple
   algebraic minimum.

Python
CopyAsk AI
# scikit-learn handles the optimization internally, but the solver choice matters
model_lbfgs = LogisticRegression(solver="lbfgs")     # default, good for most cases
model_saga = LogisticRegression(solver="saga")         # supports L1, works well for large datasets
model_liblinear = LogisticRegression(solver="liblinear")  # good for small datasets, supports L1

4. Regularization in Logistic Regression#

Unlike plain LinearRegression, scikit-learn's LogisticRegression applies L2 regularization by default — the C parameter controls its strength (note: inversely, like SVM's C, not like Ridge's alpha).

Python
CopyAsk AI
# C is the INVERSE of regularization strength
model_strong_reg = LogisticRegression(C=0.01)    # small C = STRONG regularization
model_weak_reg = LogisticRegression(C=100)          # large C = WEAK regularization

model_l1 = LogisticRegression(penalty="l1", solver="liblinear", C=1.0)   # like Lasso — can zero out coefficients
model_l2 = LogisticRegression(penalty="l2", C=1.0)                          # like Ridge — the default
model_elastic = LogisticRegression(penalty="elasticnet", solver="saga", l1_ratio=0.5, C=1.0)

This directly mirrors everything covered in the Regularization Techniques notebook — the same L1/L2/Elastic Net concepts, just applied to the log loss objective instead of squared error, and controlled through C (inverse strength) instead of alpha (direct strength).

5. Interpreting Coefficients: Odds and Log-Odds#

Logistic regression coefficients don't have the same direct "unit change in target" interpretation as linear regression — they operate on the log-odds scale.

text
Ask AICopy
log-odds = log(p / (1-p)) = w · x + b

This means: a one-unit increase in a feature changes the LOG-ODDS of the
positive class by exactly that feature's coefficient — not the
probability directly (since the sigmoid function is non-linear, the same
change in log-odds produces different changes in probability depending
on where you start).

Python
CopyAsk AI
coef = model.coef_[0][0]
odds_ratio = np.exp(coef)
print(f"Coefficient: {coef:.3f}")
print(f"Odds ratio: {odds_ratio:.3f}")
# An odds ratio of 1.5 means: a one-unit increase in this feature
# multiplies the ODDS of the positive class by 1.5 — NOT the probability
# by 1.5.

text
Ask AICopy
Odds ratio interpretation:
  odds_ratio > 1  → feature increases the odds of the positive class
  odds_ratio = 1  → feature has no effect on the odds
  odds_ratio < 1  → feature decreases the odds of the positive class

This odds-ratio interpretation is a genuinely useful and common practice in fields like medicine and social science, where logistic regression remains popular partly because of this interpretability (e.g. "each additional year of age multiplies the odds of the condition by 1.08").

6. Multiclass Classification#

Logistic regression is inherently binary; scikit-learn extends it to multiclass problems using one of two strategies.

Python
CopyAsk AI
from sklearn.datasets import load_iris

X_iris, y_iris = load_iris(return_X_y=True)   # 3 classes

# Multinomial (softmax) logistic regression — the modern default,
# generalizes the sigmoid/log-loss framework directly to multiple classes
model_multi = LogisticRegression(multi_class="multinomial", max_iter=1000)
model_multi.fit(X_iris, y_iris)

# One-vs-Rest: trains 3 separate binary classifiers, one per class
model_ovr = LogisticRegression(multi_class="ovr", max_iter=1000)
model_ovr.fit(X_iris, y_iris)

text
Ask AICopy
Multinomial (softmax): generalizes the sigmoid to multiple classes at
                          once using the softmax function, producing
                          probabilities across ALL classes that sum to 1
                          jointly — generally the preferred, more
                          statistically principled approach for genuinely
                          multiclass (not multi-label) problems.

One-vs-Rest (OvR): trains N independent binary classifiers ("this class"
                     vs "everything else"), each with its OWN sigmoid —
                     probabilities across classes are NOT guaranteed to
                     sum to 1 without additional normalization.

7. Feature Scaling and Logistic Regression#

Python
CopyAsk AI
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Especially important when using regularization (the default!) — an
# unscaled feature with a naturally large range gets penalized
# disproportionately relative to a naturally small-range feature
pipeline = make_pipeline(StandardScaler(), LogisticRegression())
pipeline.fit(X, y)

Since scikit-learn's LogisticRegression regularizes by default, feature scaling matters here for exactly the same reason covered in the Regularization Techniques notebook — always scale features before fitting, unless regularization has been deliberately disabled (penalty=None).

8. Assumptions Behind Logistic Regression#

text
Ask AICopy
1. LINEARITY OF LOG-ODDS: the log-odds of the positive class is a linear
   function of the features (not the probability itself, which is
   non-linear due to the sigmoid — this is a subtle but important
   distinction from linear regression's assumption).

2. INDEPENDENCE OF OBSERVATIONS: same as linear regression.

3. LITTLE OR NO MULTICOLLINEARITY: same reasoning and same diagnostic
   tools (VIF) as linear regression.

4. LARGE SAMPLE SIZE: maximum likelihood estimation's statistical
   properties (like the validity of standard errors) rely on
   asymptotic theory — works best with a reasonably large number of
   samples per class, especially with several features.

9. Class Imbalance and Logistic Regression#

Python
CopyAsk AI
# Logistic regression's default 0.5 threshold can perform poorly on
# imbalanced data — the model may rarely predict the minority class

model_balanced = LogisticRegression(class_weight="balanced")
# automatically re-weights the loss function to penalize misclassifying
# the minority class more heavily, inversely proportional to class
# frequency — the same underlying idea as the imbalanced-data handling
# discussed in the Model Evaluation notebook
model_balanced.fit(X, y)

Python
CopyAsk AI
# Alternative: adjust the classification threshold directly, rather than
# accepting the default 0.5
y_proba = model.predict_proba(X)[:, 1]
custom_threshold = 0.3   # lower threshold = classify more examples as positive
y_pred_custom = (y_proba >= custom_threshold).astype(int)

10. Logistic Regression vs Other Classifiers#

text
Ask AICopy
Logistic regression tends to work well when:
- The classes are (approximately) linearly separable
- Interpretability matters — coefficients translate directly to
  odds-ratio effects
- A fast, well-calibrated probability estimate is needed (logistic
  regression's probabilities tend to be better calibrated out-of-the-box
  than, say, raw SVM decision distances)
- As a strong, simple baseline before trying more complex classifiers

Logistic regression struggles when:
- The true decision boundary is highly non-linear (without substantial
  feature engineering)
- Complex feature interactions dominate the true relationship
- The dataset is very large and non-linear patterns matter — tree
  ensembles or neural networks often outperform substantially in that
  regime

Notes#

• Despite its name, logistic regression is a classification algorithm — the "regression" refers to modeling the log-odds as a linear function, not to predicting a continuous target directly.
• The sigmoid function is what converts an unbounded linear combination into a valid probability, and its smoothness is what makes gradient- based optimization of log loss possible.
• Log loss (not squared error) is used specifically because it disproportionately penalizes confident wrong predictions — a property squared error lacks.
• Coefficients operate on the log-odds scale, not the probability scale directly — the odds-ratio interpretation (exp(coefficient)) is the standard way to make them meaningful to a non-technical audience.
• scikit-learn's LogisticRegression regularizes by default (C=1.0, L2) — remember to scale features accordingly, and be aware this default behavior differs from LinearRegression, which has no built-in regularization.

Best Practices Checklist#

• Always scale features before fitting, since regularization is applied by default
• Use the odds-ratio (exp(coefficient)) interpretation when explaining feature effects to a non-technical audience
• Use multi_class="multinomial" for genuinely multiclass (not multi-label) problems rather than defaulting to One-vs-Rest
• Use class_weight="balanced" (or adjust the decision threshold directly) when working with imbalanced classes
• Check for multicollinearity via VIF when coefficient interpretation matters, not just raw predictive accuracy
• Consider feature engineering (polynomial/interaction terms) before concluding logistic regression "can't handle" a non-linear problem
• Use predict_proba() rather than just predict() when the actual probability estimate (not just the class label) is useful for downstream decisions
• Compare against tree ensembles or other non-linear classifiers as a baseline check whenever the linear decision boundary assumption seems questionable