Support Vector Machines

Overview#

Support Vector Machines (SVMs) are a powerful classification (and regression) method built on a geometric idea: find the decision boundary that separates classes with the widest possible margin. This notebook covers the maximum margin intuition, hard vs soft margins, the kernel trick for handling non-linearly separable data, SVM for regression, and practical tuning guidance.

1. The Maximum Margin Intuition#

For linearly separable data, there are infinitely many hyperplanes that could separate the two classes perfectly. SVM picks the one specific hyperplane that maximizes the margin — the distance between the decision boundary and the nearest data points from each class.

Python
CopyAsk AI
import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

X, y = make_blobs(n_samples=100, centers=2, random_state=6, cluster_std=1.2)

svm = SVC(kernel="linear", C=1000)   # very high C approximates a "hard margin"
svm.fit(X, y)

print(f"Number of support vectors: {len(svm.support_vectors_)}")

Support vectors: the only points that matter#

Python
CopyAsk AI
# The decision boundary is determined ENTIRELY by the support vectors —
# the points closest to the boundary. Every other point could be moved
# (as long as it stays on the correct side, away from the margin) without
# changing the decision boundary at all.
print(f"Support vector indices: {svm.support_}")
print(f"Support vectors:\n{svm.support_vectors_}")

This is a defining and useful property of SVMs: unlike, say, linear regression (where every data point influences the fitted line), an SVM's decision boundary depends only on the "hardest" points near the margin — the support vectors. Points far from the boundary, deep within their class's region, have zero influence on where the boundary is drawn.

2. Hard Margin vs Soft Margin#

A hard margin SVM requires perfect separation — no training points are allowed inside the margin or on the wrong side at all. This only works for perfectly linearly separable data, and is extremely sensitive to outliers or noise (a single mislabeled point can make hard-margin SVM impossible to fit).

A soft margin SVM allows some points to violate the margin (or even be misclassified), controlled by a regularization parameter C.

Python
CopyAsk AI
from sklearn.svm import SVC

# Low C: wider margin, MORE tolerance for margin violations (more regularization)
svm_soft = SVC(kernel="linear", C=0.1)

# High C: narrower margin, LESS tolerance for margin violations (less regularization)
svm_hard = SVC(kernel="linear", C=1000)

text
Ask AICopy
Low C  → prioritizes a WIDE margin, tolerates some misclassified points
          → higher bias, lower variance → less prone to overfitting

High C → prioritizes CORRECTLY classifying every training point
          → lower bias, higher variance → more prone to overfitting
          (especially with noisy data or outliers)

C in SVM plays a directly analogous role to the inverse of alpha in Ridge/Lasso regularization (covered in the Regularization Techniques notebook) — it's the same bias-variance tradeoff, expressed through a different parameter.

Visualizing the effect of C#

Python
CopyAsk AI
X_noisy, y_noisy = make_blobs(n_samples=100, centers=2, random_state=6, cluster_std=2.5)
# adding an intentional outlier
X_noisy = np.vstack([X_noisy, [0, 0]])
y_noisy = np.append(y_noisy, 1)   # a point placed right in the "wrong" region

for c in [0.01, 1, 100]:
    svm = SVC(kernel="linear", C=c)
    svm.fit(X_noisy, y_noisy)
    print(f"C={c}: {len(svm.support_vectors_)} support vectors, "
          f"training accuracy = {svm.score(X_noisy, y_noisy):.3f}")

Low C values essentially "ignore" the outlier to keep a wide, generally sensible margin; high C values distort the boundary significantly to accommodate the outlier — often at the cost of overall generalization.

3. The Kernel Trick: Handling Non-Linear Data#

Real-world data is frequently not linearly separable. The kernel trick lets SVM find non-linear decision boundaries by implicitly mapping data into a higher-dimensional space, without ever explicitly computing that transformation (which would be computationally expensive).

Python
CopyAsk AI
from sklearn.datasets import make_circles

X_circ, y_circ = make_circles(n_samples=200, noise=0.1, factor=0.3, random_state=42)
# This data is NOT linearly separable — one class forms a ring around the other

linear_svm = SVC(kernel="linear")
linear_svm.fit(X_circ, y_circ)
print(f"Linear kernel accuracy: {linear_svm.score(X_circ, y_circ):.3f}")   # poor — can't separate a ring

rbf_svm = SVC(kernel="rbf")
rbf_svm.fit(X_circ, y_circ)
print(f"RBF kernel accuracy: {rbf_svm.score(X_circ, y_circ):.3f}")   # much better

Common kernels#

Python
CopyAsk AI
# Linear: no transformation — a straight-line (or hyperplane) boundary
SVC(kernel="linear")

# Polynomial: maps data into polynomial feature combinations
SVC(kernel="poly", degree=3)

# RBF (Radial Basis Function / Gaussian): the most commonly used
# non-linear kernel, effectively measuring similarity based on distance
SVC(kernel="rbf", gamma="scale")

# Sigmoid: similar in form to a neural network activation function
SVC(kernel="sigmoid")

text
Ask AICopy
The KERNEL TRICK: instead of transforming every data point into a
higher-dimensional space explicitly (which could be extremely expensive,
or even infinite-dimensional for RBF), the kernel function computes what
the DOT PRODUCT between two points WOULD be in that higher-dimensional
space — directly, without ever actually performing the transformation.
This makes non-linear SVMs computationally tractable.

The gamma parameter (RBF kernel)#

Python
CopyAsk AI
for gamma in [0.01, 0.1, 1, 10]:
    svm = SVC(kernel="rbf", gamma=gamma, C=1)
    svm.fit(X_circ, y_circ)
    print(f"gamma={gamma}: training accuracy = {svm.score(X_circ, y_circ):.3f}, "
          f"support vectors = {len(svm.support_vectors_)}")

gamma controls how far the influence of a single training point reaches:

text
Ask AICopy
Low gamma  → each point's influence reaches FAR — smoother, simpler
              decision boundary → higher bias, lower variance
High gamma → each point's influence is very LOCALIZED — the boundary can
              wiggle tightly around individual points → lower bias,
              higher variance (risk of overfitting)

C and gamma interact — both need to be tuned together (typically via grid search) rather than independently, since their combined effect on model complexity isn't simply additive.

4. Feature Scaling Is Critical for SVM#

Python
CopyAsk AI
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# SVM computes distances/dot products between points — unscaled features
# with vastly different ranges will dominate the distance calculation,
# regardless of their actual importance
svm_pipeline = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1, gamma="scale"))
svm_pipeline.fit(X_train, y_train)

Like regularized linear models, SVM's behavior depends directly on the numeric scale of input features (through distance and dot-product calculations) — always scale features before fitting an SVM, especially with non-linear kernels.

5. Multiclass Classification with SVM#

SVM is inherently a binary classifier; scikit-learn handles multiclass problems automatically using standard strategies.

Python
CopyAsk AI
from sklearn.svm import SVC
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)   # 3 classes

svm_multiclass = SVC(kernel="rbf", decision_function_shape="ovr")   # one-vs-rest (default)
svm_multiclass.fit(X, y)

text
Ask AICopy
One-vs-Rest (OvR): trains one binary classifier per class ("this class"
                     vs "everything else"), picks the class whose
                     classifier is most confident. Requires N classifiers
                     for N classes.

One-vs-One (OvO): trains one binary classifier for EVERY PAIR of classes,
                    and combines their votes. Requires N*(N-1)/2
                    classifiers for N classes — scikit-learn's SVC uses
                    this internally by default for the underlying fitting
                    process, though decision_function_shape controls the
                    externally visible interface.

6. Support Vector Regression (SVR)#

The same maximum-margin idea extends to regression: instead of finding a boundary that separates classes, SVR finds a function where as many training points as possible fall within a margin (called epsilon) around the predicted line, while keeping the function as "flat" (simple) as possible.

Python
CopyAsk AI
from sklearn.svm import SVR
from sklearn.datasets import make_regression

X_reg, y_reg = make_regression(n_samples=200, n_features=5, noise=10, random_state=42)

svr = SVR(kernel="rbf", C=1.0, epsilon=0.1)
svr.fit(X_reg, y_reg)
print(f"R² score: {svr.score(X_reg, y_reg):.3f}")

epsilon defines a "tube" around the predicted function — errors smaller than epsilon aren't penalized at all, only errors larger than that margin contribute to the loss, making SVR somewhat robust to small amounts of noise by design.

7. Tuning SVM Hyperparameters#

Python
CopyAsk AI
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("svm", SVC()),
])

param_grid = {
    "svm__C": [0.1, 1, 10, 100],
    "svm__gamma": [0.001, 0.01, 0.1, 1],
    "svm__kernel": ["rbf", "linear"],
}

grid_search = GridSearchCV(pipeline, param_grid, cv=5, scoring="accuracy", n_jobs=-1)
grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.3f}")

Note the svm__ prefix in the parameter grid — this is scikit-learn's convention for tuning parameters of a specific named step inside a Pipeline, in the form <step_name>__<parameter_name>.

8. When to Use (and Not Use) SVM#

text
Ask AICopy
SVM tends to work well when:
- The number of features is large relative to the number of samples
- A clear margin of separation is expected to exist between classes
- The dataset is small to medium-sized (SVM training scales poorly to
  very large datasets — often impractically slow beyond tens of
  thousands of samples with non-linear kernels)

SVM is usually NOT the first choice when:
- The dataset is very large (tree ensembles or linear models typically
  scale better and train much faster)
- Probabilistic outputs are needed by default (SVM gives distances from
  the boundary, not calibrated probabilities, though
  probability=True can enable a slower, approximate probability estimate)
- Interpretability is a priority (unlike a single decision tree, an SVM's
  decision boundary — especially with an RBF kernel — is difficult to
  explain intuitively)

Notes#

• The decision boundary in SVM depends only on the support vectors — the points closest to the margin — not on every training point, unlike most other models covered in this course.
• C controls the same bias-variance tradeoff as regularization strength elsewhere in this course, just expressed inversely (high C = less regularization, low C = more regularization).
• The kernel trick makes non-linear decision boundaries computationally feasible without explicitly transforming data into higher-dimensional space.
• C and gamma (for RBF) must be tuned together — their effects on model complexity interact, so grid search over both jointly is standard practice.
• SVM training complexity scales poorly with the number of samples (particularly for non-linear kernels), making it less suitable than tree ensembles or linear models for very large datasets.

Best Practices Checklist#

• Always scale features before fitting an SVM, since it relies on distance/dot-product calculations
• Start with a linear kernel for high-dimensional data, and only move to RBF/polynomial kernels if linear separation genuinely fails
• Tune C and gamma together via grid search, not independently
• Use a soft margin (moderate C) rather than attempting a hard margin on real-world, noisy data
• Be cautious applying SVM to very large datasets — consider tree ensembles or linear models instead if training time becomes impractical
• Use probability=True only when calibrated probability estimates are genuinely needed, since it adds computational overhead
• Inspect the number of support vectors as a rough proxy for model complexity — a very high proportion can signal overfitting or a poorly chosen C/gamma