Notebook 4: Logistic Regression and Classification Basics

Module: Machine Learning Level: Beginner to Intermediate Prerequisites: Notebook 3 (Linear Regression and Regularization) Estimated Time: 3–4 hours

---

Table of Contents#

1. From Regression to Classification
2. Why Not Just Use Linear Regression for Classification
3. The Sigmoid Function
4. The Logistic Regression Model
5. The Loss Function: Log Loss / Cross-Entropy
6. Fitting Logistic Regression: Gradient Descent
7. Decision Boundaries
8. Interpreting Coefficients: Odds and Log-Odds
9. Regularized Logistic Regression
10. Multiclass Classification Strategies
11. Hands-On: Binary and Multiclass Classification
12. Case Studies
13. Common Misconceptions and Pitfalls
14. Glossary
15. Exercises
16. Quiz
17. Summary
18. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Explain why linear regression is unsuitable for classification and how logistic regression addresses this
• Derive the sigmoid function and explain its role in producing valid probability outputs
• Write and interpret the log loss (cross-entropy) function for binary classification
• Explain, at a conceptual level, how gradient descent fits a logistic regression model
• Interpret logistic regression coefficients in terms of odds and log-odds
• Apply L1 and L2 regularization to logistic regression
• Choose and apply an appropriate multiclass classification strategy (One-vs-Rest, One-vs-One, Softmax)
• Fit and evaluate binary and multiclass logistic regression models in scikit-learn

---

1. From Regression to Classification#

Notebook 3 dealt with predicting a continuous target $y \in \mathbb{R}$. Many real problems instead require predicting a category: will this customer churn or not, is this tumor malignant or benign, which of five product categories does this item belong to. This notebook introduces logistic regression, the classification analogue of linear regression, and the foundation for understanding the more complex classifiers in Notebooks 6 through 8.

Despite its name, logistic regression is a classification algorithm, not a regression algorithm in the sense of Notebook 3 — the name is a historical artifact from its origins in statistics, where it was developed to model probabilities as a function of predictors.

---

2. Why Not Just Use Linear Regression for Classification#

2.1 The Naive Approach#

Suppose $y \in \{0, 1\}$ (a binary label) and we attempt to fit ordinary linear regression directly: $\hat{y} = \theta_0 + \theta_1 x$. Two serious problems arise immediately.

2.2 Problem 1: Unbounded Output#

Linear regression can output any real number, including values far below 0 or above 1. But if $\hat{y}$ is meant to represent a probability of belonging to class 1, values like -0.4 or 1.8 are meaningless.

2.3 Problem 2: Sensitivity to Outlying Feature Values#

Consider a dataset predicting whether a tumor is malignant (1) or benign (0) based on tumor size. If one additional data point with an extremely large tumor size (still labeled malignant) is added, a fitted straight line will tilt to accommodate it, which can shift the decision threshold (where the line crosses 0.5) for all the other, unrelated data points — a straight line is a poor fit to a fundamentally bounded, saturating relationship between a predictor and a probability.

2.4 What We Actually Need#

We need a function that:

1. Always outputs a value between 0 and 1, so it can be interpreted as a probability
2. Is monotonic (as the linear combination of features increases, the probability should move consistently in one direction)
3. Saturates at the extremes (very large or very small linear combinations should map close to, but never exactly, 0 or 1)

The sigmoid function, introduced next, satisfies all three properties.

---

3. The Sigmoid Function#

3.1 Definition#

$\sigma(z) = \frac{1}{1 + e^{-z}}$

where $z$ can be any real number.

3.2 Key Properties#

• $\sigma(0) = 0.5$
• As $z \to +\infty$, $\sigma(z) \to 1$
• As $z \to -\infty$, $\sigma(z) \to 0$
• $\sigma(z)$ is monotonically increasing in $z$
• The output is always strictly between 0 and 1, never exactly reaching either bound

3.3 Shape#

code
Ask AICopy
 sigma(z)
   1.0 |                        _______________
       |                   ___/
       |                _/
   0.5 |             _/
       |          _/
       |      __/
   0.0 |___/____________________________________
       -6   -4   -2    0    2    4    6      z

This characteristic "S" shape (hence "sigmoid") gives the function its saturating behavior at the extremes while transitioning smoothly through 0.5 at $z=0$.

3.4 Code Example#

Python
CopyAsk AI
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z_values = np.array([-6, -2, 0, 2, 6])
print(np.round(sigmoid(z_values), 4))

Expected output:

[0.0025 0.1192 0.5    0.8808 0.9975]

---

4. The Logistic Regression Model#

4.1 The Model Definition#

Logistic regression first computes a linear combination of features, exactly as in Notebook 3:

$z = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \dots + \theta_d x_d = \theta^T x$

and then passes $z$ through the sigmoid function to produce a probability:

$\hat{y} = P(y=1 \mid x; \theta) = \sigma(\theta^T x) = \frac{1}{1 + e^{-\theta^T x}}$

4.2 From Probability to a Hard Prediction#

To produce a class label rather than a probability, apply a threshold, conventionally 0.5:

$\hat{y}_{\text{class}} = \begin{cases} 1 & \text{if } \hat{y} \geq 0.5 \\ 0 & \text{if } \hat{y} < 0.5 \end{cases}$

This threshold is itself a tunable choice, not a fixed law — Notebook 11 (Advanced Model Evaluation) covers how shifting this threshold trades off precision against recall depending on the relative costs of different error types.

---

5. The Loss Function: Log Loss / Cross-Entropy#

5.1 Why Squared Error Is a Poor Fit Here#

If we tried to use squared error loss (Notebook 3, Section 2.2) with a sigmoid-based model, the resulting cost function would be non-convex in $\theta$, meaning gradient descent could get stuck in local minima rather than finding the true best solution. Logistic regression instead uses a loss function specifically derived from probability theory that remains convex for this model.

5.2 The Log Loss Formula#

For a single example, log loss (also called binary cross-entropy) is defined as:

$L(y, \hat{y}) = -\big[y \log(\hat{y}) + (1-y)\log(1-\hat{y})\big]$

This can be understood piecewise:

• If the true label $y = 1$: the loss simplifies to $-\log(\hat{y})$. This is small when $\hat{y}$ is close to 1 (a confident, correct prediction) and grows without bound as $\hat{y} \to 0$ (a confident, wrong prediction is punished severely).
• If the true label $y = 0$: the loss simplifies to $-\log(1-\hat{y})$. This is small when $\hat{y}$ is close to 0, and grows without bound as $\hat{y} \to 1$.

5.3 Why the Asymmetric, Unbounded Penalty Matters#

Log loss's key behavior is that it penalizes confident, wrong predictions extremely heavily (approaching infinite loss as a confidently wrong prediction approaches certainty), while a merely inaccurate but appropriately uncertain prediction (e.g., predicting 0.6 when the true label is 0) receives only a moderate penalty. This directly encourages the model to be well-calibrated: confident only when the evidence genuinely supports confidence.

5.4 The Full Cost Function#

Following the Empirical Risk Minimization framework from Notebook 0, Section 4.4, the total cost across $n$ training examples is:

$J(\theta) = -\frac{1}{n}\sum_{i=1}^{n} \Big[y^{(i)} \log(\hat{y}^{(i)}) + (1-y^{(i)})\log(1-\hat{y}^{(i)})\Big]$

where $\hat{y}^{(i)} = \sigma(\theta^T x^{(i)})$. Unlike OLS in Notebook 3, this function has no closed-form solution and must be minimized iteratively, as covered next.

---

6. Fitting Logistic Regression: Gradient Descent#

6.1 Why No Closed-Form Solution Exists#

Setting the gradient of $J(\theta)$ to zero, as was done for OLS in Notebook 3, Section 3.2, produces a system of nonlinear equations (due to the sigmoid function) with no algebraic closed-form solution. Instead, logistic regression is fit using iterative optimization.

6.2 The Gradient Descent Update Rule#

Remarkably, despite the added complexity of the sigmoid function, the gradient of the log loss with respect to $\theta$ takes a strikingly similar form to the gradient for linear regression:

$\nabla_\theta J(\theta) = \frac{1}{n}\sum_{i=1}^{n} \big(\hat{y}^{(i)} - y^{(i)}\big) x^{(i)}$

The gradient descent update rule is then:

$\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta)$

where $\alpha$ is the learning rate introduced in Notebook 3, Section 3.4. Each step nudges $\theta$ in the direction that most reduces prediction error, weighted by how confidently wrong each prediction currently is.

6.3 Convergence#

Because $J(\theta)$ is convex for logistic regression, gradient descent (with an appropriately chosen learning rate) is guaranteed to converge to the global minimum, unlike optimization for many of the more complex, non-convex models introduced later in this course (such as neural networks, outside this course's scope, though tree ensembles in Notebook 6 face related non-convexity considerations in their own way).

6.4 In Practice#

scikit-learn's LogisticRegression does not require you to implement gradient descent manually; it uses efficient solvers (such as lbfgs, a quasi-Newton method, by default) that converge faster than plain gradient descent for most practical dataset sizes. Understanding the gradient descent mechanics above remains valuable because the same underlying principles reappear in more advanced models outside classical ML.

---

7. Decision Boundaries#

7.1 Definition#

A decision boundary is the surface in feature space where the model's predicted probability equals exactly 0.5, i.e., where $\theta^T x = 0$. On one side of this boundary, the model predicts class 1; on the other, class 0.

7.2 Linear Decision Boundaries#

Because $z = \theta^T x$ is a linear function of the features, the decision boundary of logistic regression is always a straight line (in two dimensions), a flat plane (in three dimensions), or a flat hyperplane (in higher dimensions) — regardless of how the sigmoid curves the probability output. This is an important and sometimes counterintuitive point: the probability surface is smoothly curved, but the boundary that separates predicted classes is perfectly flat.

Architecture & Data Flow
Copy
Feature 2
    |
    |     x  x  x           x = Class 1
    |    x  x   \            o = Class 0
    |   x    x   \  <- decision boundary (theta^T x = 0)
    |  o   o     \
    | o  o   o    \
    |o    o        \
    +------------------------------ Feature 1

7.3 When Linear Boundaries Are Insufficient#

If the true classes are not linearly separable (for example, one class forms a ring around the other), plain logistic regression cannot separate them well, regardless of how much data is provided, since it is fundamentally restricted to the hypothesis space of linear boundaries. Section 6.1 in Notebook 7 revisits this exact limitation when introducing kernel methods, and polynomial or interaction features (Notebook 2, Section 6.1) can also be added to logistic regression itself to create curved effective boundaries in the original feature space.

---

8. Interpreting Coefficients: Odds and Log-Odds#

8.1 Odds#

The odds of an event is the ratio of the probability it happens to the probability it does not:

$\text{odds} = \frac{P(y=1)}{P(y=0)} = \frac{\hat{y}}{1 - \hat{y}}$

8.2 Log-Odds (Logit)#

Taking the natural log of the odds recovers exactly the linear combination $z$ computed before applying the sigmoid — this is not a coincidence, but the defining algebraic relationship of logistic regression:

$\log\left(\frac{\hat{y}}{1-\hat{y}}\right) = \theta_0 + \theta_1 x_1 + \dots + \theta_d x_d$

This is why the function $\sigma^{-1}$ (the inverse of the sigmoid) is called the logit function, and why logistic regression is sometimes described as "linear regression on the log-odds."

8.3 Interpreting a Coefficient#

Because the model is linear in log-odds, a one-unit increase in $x_j$ (holding other features constant, exactly as in Notebook 3, Section 4.2) increases the log-odds of the outcome by exactly $\theta_j$. Exponentiating $\theta_j$ gives the multiplicative effect on the odds themselves:

$\text{odds ratio} = e^{\theta_j}$

For example, if $\theta_j = 0.7$ for a feature "years of smoking," then $e^{0.7} \approx 2.01$, meaning each additional year of smoking is associated with the odds of the outcome (say, developing a disease) roughly doubling, holding other features constant.

8.4 Code Example#

Python
CopyAsk AI
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X, y = data.data, data.target

model = LogisticRegression(max_iter=5000)
model.fit(X, y)

odds_ratios = np.exp(model.coef_[0])
top_feature_idx = np.argmax(np.abs(model.coef_[0]))
print(f"Feature with largest effect: {data.feature_names[top_feature_idx]}")
print(f"Odds ratio: {odds_ratios[top_feature_idx]:.3f}")

---

9. Regularized Logistic Regression#

9.1 Why Regularize#

Exactly as in Notebook 3, logistic regression can overfit, particularly with many features relative to observations, or with highly correlated features. scikit-learn's LogisticRegression applies L2 regularization by default.

9.2 The Regularized Cost Function#

$J_{\text{regularized}}(\theta) = J(\theta) + \lambda \sum_{j=1}^{d} \theta_j^2 \quad \text{(L2, "Ridge-style")}$

$J_{\text{regularized}}(\theta) = J(\theta) + \lambda \sum_{j=1}^{d} |\theta_j| \quad \text{(L1, "Lasso-style")}$

exactly paralleling Notebook 3, Sections 8 and 9. The key practical difference is scikit-learn's parameterization: LogisticRegression uses C, the inverse of regularization strength, so smaller C means stronger regularization (the opposite direction from alpha in Ridge/Lasso).

9.3 Code Example#

Python
CopyAsk AI
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Strong L1 regularization for automatic feature selection (Notebook 3, Section 9.3)
model = make_pipeline(
    StandardScaler(),
    LogisticRegression(penalty="l1", C=0.1, solver="liblinear", max_iter=5000)
)
model.fit(X_train, y_train)

---

10. Multiclass Classification Strategies#

10.1 The Problem#

Logistic regression, as derived above, is inherently binary: it models $P(y=1 \mid x)$ versus $P(y=0 \mid x)$. Many real problems have more than two classes (e.g., classifying an image as a cat, dog, or bird). Three strategies extend binary classifiers to the multiclass setting.

10.2 One-vs-Rest (OvR)#

Train $K$ separate binary classifiers, one per class, where classifier $k$ is trained to distinguish class $k$ from all other classes combined. At prediction time, run all $K$ classifiers and choose the class whose classifier outputs the highest probability.

code
Ask AICopy
Classifier 1: "Is it class A?" vs "Is it not class A?"
Classifier 2: "Is it class B?" vs "Is it not class B?"
Classifier 3: "Is it class C?" vs "Is it not class C?"
Final prediction: argmax of the three probabilities

Advantage: simple, only requires $K$ classifiers. Disadvantage: each binary classifier is trained on an imbalanced problem (one class versus everything else), and the probabilities from separately trained classifiers are not guaranteed to be on a truly comparable scale.

10.3 One-vs-One (OvO)#

Train a separate binary classifier for every pair of classes, resulting in $\binom{K}{2} = \frac{K(K-1)}{2}$ classifiers. At prediction time, each classifier "votes" for one of its two classes, and the class with the most votes wins.

Advantage: each individual classifier only ever sees two classes' worth of data, often a more balanced problem. Disadvantage: the number of classifiers grows quadratically with $K$, making OvO expensive for problems with many classes.

10.4 Softmax Regression (Multinomial Logistic Regression)#

Rather than combining multiple binary classifiers, softmax regression directly generalizes logistic regression to $K$ classes in a single, unified model. It computes a separate linear score $z_k = \theta_k^T x$ for each class, then converts all $K$ scores into a valid probability distribution using the softmax function:

$P(y=k \mid x) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}$

Notice that when $K=2$, softmax reduces algebraically to the sigmoid function from Section 3, confirming binary logistic regression as a special case of this more general model.

Advantage: produces one coherent, properly normalized probability distribution across all classes in a single training procedure, generally the preferred default for multiclass logistic regression today. Disadvantage: slightly more complex to derive and, unlike OvR/OvO, cannot be trivially assembled from independently trained binary classifiers.

10.5 Comparison Table#

---

11. Hands-On: Binary and Multiclass Classification#

11.1 Binary Classification With Interpretation#

Python
CopyAsk AI
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = LogisticRegression(max_iter=5000)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)

print(f"Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print(f"Confusion Matrix:\n{confusion_matrix(y_test, y_pred)}")
print(f"First 5 predicted probabilities:\n{y_proba[:5]}")

Expected output (approximate):

Mathematical Formulation
Explain FormulaCopy
Accuracy: 97.37%
Confusion Matrix:
[[41  1]
 [ 2 70]]
First 5 predicted probabilities:
[[0.99 0.01]
 [0.02 0.98]
 [0.95 0.05]
 [0.01 0.99]
 [0.87 0.13]]

Note: predict_proba returns two columns for binary classification — the probability of class 0 and the probability of class 1, summing to 1 in each row. Confusion matrices are covered in depth in Notebook 11.

11.2 Multiclass Classification With Softmax#

Python
CopyAsk AI
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

data = load_iris()
X, y = data.data, data.target  # 3 classes

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# multi_class="multinomial" explicitly requests softmax regression (Section 10.4)
model = LogisticRegression(multi_class="multinomial", max_iter=5000)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print(f"Predicted probabilities for first test example: {model.predict_proba(X_test)[0]}")
print(f"Classes: {model.classes_}")

Expected output (approximate):

code
Ask AICopy
Accuracy: 100.00%
Predicted probabilities for first test example: [0.001 0.089 0.910]
Classes: [0 1 2]

The three probabilities for the first test example sum to 1, confirming the softmax normalization property described in Section 10.4.

---

12. Case Studies#

12.1 Case Study: Medical Screening Threshold Adjustment#

A hospital deploys a logistic regression model to flag patients at risk of a rare but serious condition, using the default 0.5 threshold from Section 4.2. Very few patients are flagged, and several known cases are missed.

Diagnosis: Because the condition is rare, the model's predicted probabilities for true positive cases may often fall below 0.5 even when the model has correctly identified elevated risk relative to the general population; the model is not necessarily wrong, but the arbitrary 0.5 threshold is poorly suited to a screening context where missing a case (a false negative) is far more costly than a false alarm (a false positive).

Lesson: The threshold in Section 4.2 is a business decision, not a mathematical requirement. Lowering the threshold (for example, to 0.2) trades additional false positives for fewer false negatives, an appropriate tradeoff in high-stakes screening contexts — Notebook 11 formalizes this tradeoff using precision-recall curves.

12.2 Case Study: Credit Scoring and Odds Ratio Interpretation#

A bank's logistic regression credit model produces a coefficient of -0.05 for "number of recent credit inquiries." Regulators require the bank to explain, in plain language, the effect of this feature.

Diagnosis: Using Section 8.3, the odds ratio is $e^{-0.05} \approx 0.951$, meaning each additional recent credit inquiry is associated with roughly a 4.9 percent decrease in the odds of loan approval, holding other factors constant. This odds-ratio interpretation, directly available due to logistic regression's log-odds linearity (Section 8.2), is a major reason logistic regression remains popular in regulated industries like banking and insurance, despite more complex models (Notebooks 6-8) often achieving higher raw accuracy — regulatory and legal requirements often demand exactly this kind of transparent, coefficient-level explanation.

Lesson: Model choice is not purely about predictive performance; in regulated or high-stakes domains, the interpretability advantages of logistic regression's odds ratios can outweigh the accuracy gains of more complex, less transparent alternatives.

12.3 Case Study: One-vs-Rest Struggling With Imbalanced Multiclass Data#

A team building a support ticket categorization system (5 categories, one of which represents 70 percent of tickets) uses One-vs-Rest logistic regression (Section 10.2) and finds that the minority categories are frequently misclassified as the majority category.

Diagnosis: Under OvR, the binary classifier for each minority category is trained on a highly imbalanced "this category vs. everything else" problem, where "everything else" is dominated by the majority category, per the disadvantage noted in Section 10.2. The classifier can achieve deceptively low training loss by leaning toward predicting "not this minority category" most of the time.

Lesson: Switching to softmax regression (Section 10.4), which jointly models all categories in a single normalized distribution, combined with class-weighting (assigning higher loss penalty to minority-class errors, a technique covered further in Notebook 11), resolved the imbalance issue considerably better than the OvR approach.

---

13. Common Misconceptions and Pitfalls#

---

14. Glossary#

---

15. Exercises#

Exercise 1 (Conceptual)#

Explain, using the shape of the sigmoid function (Section 3.3), why logistic regression is relatively robust to a single extreme outlier in a feature value, unlike linear regression fit directly to a 0/1 target (Section 2.3).

<details> <summary>Solution</summary>

Because the sigmoid saturates as its input moves far from zero (Section 3.2), an extremely large or small value of $z = \theta^Tx$ for an outlying data point produces a predicted probability very close to, but never exactly at, 0 or 1, without requiring the linear combination itself to become extreme in magnitude to explain the outcome. A plain linear regression fit directly to a 0/1 target has no such saturation and must tilt its single straight line to accommodate the outlier's raw feature value, which then distorts predictions for every other point, as described in Section 2.3.

</details>

---

Exercise 2 (Conceptual)#

A logistic regression model trained on a marketing dataset has a coefficient of 0.4 on a binary feature "received_promotional_email" (1 if yes, 0 if no). Compute the odds ratio and explain what it means in plain language.

<details> <summary>Solution</summary>

Using Section 8.3, the odds ratio is $e^{0.4} \approx 1.49$. This means that, holding other features constant, customers who received the promotional email have odds of the outcome (e.g., making a purchase) that are approximately 49 percent higher than customers who did not receive the email.

</details>

---

Exercise 3 (Coding)#

Using the load_wine dataset, fit a softmax (multinomial) logistic regression model and print the predicted probability distribution for the first three test examples, verifying that each row sums to 1.

<details> <summary>Solution</summary>

Python
CopyAsk AI
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np

data = load_wine()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = LogisticRegression(multi_class="multinomial", max_iter=5000)
model.fit(X_train, y_train)

probas = model.predict_proba(X_test)[:3]
print(probas)
print(f"Row sums: {np.round(probas.sum(axis=1), 4)}")
# Row sums should print as [1. 1. 1.], confirming the softmax
# normalization property described in Section 10.4.

</details>

---

Exercise 4 (Applied Thinking)#

You are building a spam filter and must choose a probability threshold (Section 4.2, 12.1) for flagging an email as spam. Describe the tradeoff involved in setting the threshold very low (e.g., 0.1) versus very high (e.g., 0.9), in terms of the two types of classification errors.

<details> <summary>Solution</summary>

A very low threshold (0.1) means the model flags an email as spam even when it is only mildly confident, which will catch more true spam (fewer false negatives, i.e., fewer spam emails reaching the inbox) but will also incorrectly flag more legitimate emails as spam (more false positives). A very high threshold (0.9) requires very high confidence before flagging spam, reducing false positives (fewer legitimate emails wrongly filtered) but allowing more true spam through undetected (more false negatives). The right choice depends on the relative cost of each error type in this application — for most users, a false positive (losing a legitimate email) is considered more costly than a false negative (seeing an extra spam email), which is why most production spam filters are tuned toward a relatively high threshold. This exact tradeoff is formalized using precision and recall in Notebook 11.

</details>

---

16. Quiz#

Q1. What is the range of possible outputs from the sigmoid function? a) Any real number b) Exactly 0 or 1 c) Strictly between 0 and 1 d) Between -1 and 1

Q2. Why is squared error loss not used for logistic regression? a) It is too computationally expensive b) It produces a non-convex cost function when combined with the sigmoid, risking local minima c) It only works for multiclass problems d) It cannot be differentiated

Q3. What shape is the decision boundary of standard logistic regression, without added polynomial or interaction features? a) Always curved, matching the sigmoid's shape b) Always linear/flat, regardless of the sigmoid c) Depends on the value of the learning rate d) Always circular

Q4. Which multiclass strategy trains a single unified model that directly outputs a normalized probability distribution across all classes? a) One-vs-Rest b) One-vs-One c) Softmax (multinomial) regression d) None of the above; all classifiers are inherently binary

Q5. True or False: In scikit-learn's LogisticRegression, a smaller value of C corresponds to weaker regularization.

<details> <summary>Answers</summary>

1. c) Strictly between 0 and 1
2. b) It produces a non-convex cost function when combined with the sigmoid, risking local minima
3. b) Always linear/flat, regardless of the sigmoid, per Section 7.2
4. c) Softmax (multinomial) regression
5. False — C is the inverse of regularization strength, so a smaller C means stronger regularization, per Section 9.2

</details>

---

17. Summary#

• Logistic regression extends the linear modeling framework from Notebook 3 to classification by passing a linear combination of features through the sigmoid function, producing a valid probability.
• Log loss (cross-entropy), not squared error, is used to train logistic regression, since it remains convex and appropriately penalizes confident, incorrect predictions.
• No closed-form solution exists for logistic regression; it is fit using iterative methods such as gradient descent, following an update rule structurally similar to that of linear regression.
• The decision boundary of logistic regression is always linear in the original feature space, regardless of the sigmoid's curvature.
• Coefficients have a clean, interpretable meaning in terms of log-odds and odds ratios, which is a major reason logistic regression remains preferred in regulated, interpretability-sensitive domains.
• L1 and L2 regularization apply to logistic regression exactly as they do to linear regression, controlled in scikit-learn by the inverse-strength parameter C.
• Three main strategies (One-vs-Rest, One-vs-One, and softmax regression) extend binary classification to multiple classes, with softmax generally preferred as the modern default for logistic regression specifically.

---

18. Further Reading#

• Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements of Statistical Learning, Chapter 4 (Linear Methods for Classification)
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning, Chapter 4
• scikit-learn documentation on logistic regression: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
• scikit-learn documentation on multiclass strategies: https://scikit-learn.org/stable/modules/multiclass.html

---

Next Up#

Notebook 5: Basic Model Evaluation — accuracy, precision, recall, F1-score, and the fundamental regression metrics (MSE, RMSE, R-squared), giving you the tools to properly judge the regression and classification models built in Notebooks 3 and 4 before moving to more advanced algorithms.

---

Course: Machine Learning | Notebook 4 of 13