Dimensionality Reduction

Overview#

High-dimensional data — many features per sample — creates problems: it's hard to visualize, suffers from the curse of dimensionality (covered in the KNN notebook), and often contains redundant or correlated information. Dimensionality reduction techniques compress data into fewer dimensions while preserving as much meaningful structure as possible. This notebook covers Principal Component Analysis (PCA), t-SNE, and the important distinction between feature extraction and feature selection.

1. Feature Extraction vs Feature Selection#

Both reduce the number of features a model sees, but in fundamentally different ways.

text
Ask AICopy
Feature SELECTION: choose a SUBSET of the original features, discarding
                     the rest entirely. The remaining features keep their
                     original meaning.
                     Example: keeping "age" and "income", dropping
                     "zip_code" — age and income are still exactly what
                     they were before.

Feature EXTRACTION: create NEW features that are combinations of the
                      original ones, typically fewer in number. The new
                      features generally don't have a directly
                      interpretable real-world meaning anymore.
                      Example: PCA's "principal component 1" might be a
                      weighted mix of age, income, and five other
                      features — not any single interpretable quantity.

Lasso regression (covered in the Regularization notebook) is a form of feature selection — it zeroes out coefficients, effectively dropping features while keeping the rest interpretable. PCA and t-SNE (this notebook) perform feature extraction instead.

2. Principal Component Analysis (PCA): The Core Idea#

PCA finds new axes (principal components) along which the data varies the most, and re-expresses the data in terms of these new axes — ordered so the first component captures the most variance, the second captures the next most (while being orthogonal/uncorrelated with the first), and so on.

Python
CopyAsk AI
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

# PCA is sensitive to feature scale — always standardize first
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca = PCA(n_components=2)   # reduce from 4 features down to 2
X_pca = pca.fit_transform(X_scaled)

print(f"Original shape: {X_scaled.shape}")     # (150, 4)
print(f"Reduced shape: {X_pca.shape}")           # (150, 2)

Explained variance: how much information was kept#

Python
CopyAsk AI
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance retained: {pca.explained_variance_ratio_.sum():.3f}")

›Explained variance ratio: [0.729 0.229]
Total variance retained: 0.958

This means the first two principal components together capture about 95.8% of the total variance present in the original 4 features — a strong compression with relatively little information loss, in this particular case.

Choosing the number of components#

Python
CopyAsk AI
pca_full = PCA().fit(X_scaled)   # keep all components initially, to inspect

cumulative_variance = np.cumsum(pca_full.explained_variance_ratio_)
print(cumulative_variance)   # [0.729 0.958 0.995 1.000]

# A common rule of thumb: keep enough components to explain, e.g., 95% of variance
n_components_95 = np.argmax(cumulative_variance >= 0.95) + 1
print(f"Components needed for 95% variance: {n_components_95}")

Python
CopyAsk AI
# Alternative: specify the target variance directly, let PCA determine n_components
pca_auto = PCA(n_components=0.95)   # automatically picks enough components for 95% variance
X_pca_auto = pca_auto.fit_transform(X_scaled)
print(f"Number of components chosen: {pca_auto.n_components_}")

The scree plot#

Python
CopyAsk AI
import matplotlib.pyplot as plt

plt.plot(range(1, len(pca_full.explained_variance_ratio_) + 1),
          pca_full.explained_variance_ratio_, marker="o")
plt.xlabel("Principal Component")
plt.ylabel("Explained Variance Ratio")
plt.title("Scree Plot")
plt.show()

A scree plot shows explained variance per component — similar in spirit to the elbow method for K-Means (covered in the Clustering notebook), looking for the point where additional components stop contributing much.

3. How PCA Works, Conceptually#

text
Ask AICopy
1. Center the data (subtract the mean of each feature) — and typically
   scale it too, so features with larger raw ranges don't dominate.
2. Compute the covariance matrix, capturing how every pair of features
   varies together.
3. Find the EIGENVECTORS of the covariance matrix — these are the
   principal component directions — and their corresponding
   EIGENVALUES, which indicate how much variance lies along each
   direction.
4. Sort components by eigenvalue (descending) — the direction with the
   most variance becomes principal component 1, the next-most becomes
   component 2, and so on.
5. Project the original data onto the top k components to get the
   reduced representation.

Python
CopyAsk AI
# Inspecting the actual component directions (loadings)
print("Component loadings (which original features each PC is made of):")
print(pca.components_)
# Each row is a principal component; each column's value shows how much
# that original feature contributes to that component

text
Ask AICopy
Component loadings:
[[ 0.52 -0.27  0.58  0.56]     <- PC1: mostly petal length/width + sepal length
 [ 0.37  0.92  0.02  0.06]]     <- PC2: mostly sepal width

4. Reconstructing Data from Principal Components#

Python
CopyAsk AI
# PCA is reversible (with some information loss if n_components < original)
X_reconstructed = pca.inverse_transform(X_pca)

reconstruction_error = np.mean((X_scaled - X_reconstructed) ** 2)
print(f"Mean reconstruction error: {reconstruction_error:.4f}")

The reconstruction error quantifies exactly how much information was lost by reducing to fewer dimensions — a useful sanity check, and the basis for using PCA in some anomaly detection approaches (points that reconstruct poorly are unusual relative to the learned principal components).

5. PCA for Visualization#

One of PCA's most common uses is compressing high-dimensional data down to 2 or 3 dimensions purely to make it visualizable — since humans can't directly perceive a 10-dimensional scatter plot.

Python
CopyAsk AI
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)

plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, cmap="viridis")
plt.xlabel(f"PC1 ({pca_2d.explained_variance_ratio_[0]:.1%} variance)")
plt.ylabel(f"PC2 ({pca_2d.explained_variance_ratio_[1]:.1%} variance)")
plt.title("Iris dataset projected onto 2 principal components")
plt.show()

6. PCA for Preprocessing Before Modeling#

Python
CopyAsk AI
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("pca", PCA(n_components=0.95)),
    ("classifier", LogisticRegression(max_iter=1000)),
])
pipeline.fit(X_train, y_train)

Reducing dimensionality before feeding data into a downstream model can: speed up training (fewer features to process), reduce overfitting risk (fewer parameters relative to sample size), and remove multicollinearity issues (principal components are, by construction, uncorrelated with each other) — though it can also discard some genuinely useful signal, so it's not automatically an improvement and should be validated with cross-validation.

7. t-SNE: Non-Linear Dimensionality Reduction for Visualization#

PCA finds linear combinations of features — it can miss non-linear structure in the data entirely. t-SNE (t-distributed Stochastic Neighbor Embedding) is designed specifically for visualization, and excels at revealing non-linear cluster structure that PCA would flatten away.

Python
CopyAsk AI
from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_tsne = tsne.fit_transform(X_scaled)

plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap="viridis")
plt.title("t-SNE visualization")
plt.show()

Key differences from PCA#

text
Ask AICopy
PCA:
- Linear technique — preserves GLOBAL structure and distances reasonably well
- Deterministic — same input always gives the same output
- Fast, scales well to large datasets
- Components have a mathematical meaning (variance-maximizing directions)
- Can be applied to NEW, unseen data after fitting (transform())

t-SNE:
- Non-linear technique — excels at preserving LOCAL structure (which
  points are near each other) but can distort or mislead about global
  distances between distant clusters
- Stochastic — different runs (or even different `random_state` values)
  can produce visibly different layouts
- Slow, scales poorly to very large datasets
- Output dimensions have NO interpretable meaning at all
- Cannot easily be applied to new data — has to be refit on the combined
  dataset (no natural transform() for unseen points)

The perplexity parameter#

Python
CopyAsk AI
for perplexity in [5, 30, 50]:
    tsne = TSNE(n_components=2, perplexity=perplexity, random_state=42)
    X_tsne = tsne.fit_transform(X_scaled)
    # different perplexity values can produce meaningfully different-looking
    # cluster layouts — it's worth trying a few values rather than
    # trusting a single run blindly

perplexity roughly controls the balance between preserving local versus more global structure — think of it as loosely related to the expected number of close neighbors for each point. There's no single "correct" value; typical guidance is to try values between 5 and 50 and compare.

A critical warning about t-SNE#

text
Ask AICopy
NEVER interpret the size of a cluster or the distance BETWEEN separate
clusters in a t-SNE plot as meaningful — t-SNE only reliably preserves
which points are LOCALLY close to each other. Two clusters that appear
far apart in a t-SNE plot are not necessarily more "different" than two
clusters that appear closer together; cluster SIZES in the plot also
don't reliably reflect the actual spread of the underlying data.

This is one of the most common misinterpretations of t-SNE visualizations in practice — it's a tool for spotting that clusters exist and roughly which points group together, not for making quantitative claims about cluster separation or size from the plot alone.

8. Other Dimensionality Reduction Techniques (Brief Mention)#

Python
CopyAsk AI
# UMAP: similar goals to t-SNE, often faster and better at preserving
# some global structure — a common modern alternative
# pip install umap-learn
# from umap import UMAP
# reducer = UMAP(n_components=2, random_state=42)
# X_umap = reducer.fit_transform(X_scaled)

# LDA (Linear Discriminant Analysis): unlike PCA, this is SUPERVISED —
# it finds axes that best SEPARATE known classes, rather than axes that
# simply maximize variance
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X_scaled, y)   # note: LDA requires y, PCA does not

LDA is worth knowing about specifically because, unlike everything else in this notebook, it uses label information — making it a supervised dimensionality reduction technique, generally more effective than PCA when the downstream goal is specifically classification.

9. PCA vs t-SNE: When to Use Which#

Notes#

• PCA requires feature scaling beforehand, exactly like KNN and SVM — without it, features with larger raw scales dominate the variance calculation regardless of their real importance.
• The principal components themselves generally lose direct real-world interpretability — "PC1" is a mathematical construct, not a named, meaningful quantity, unlike features kept via feature selection.
• t-SNE is a visualization tool, not a general-purpose preprocessing step — its output shouldn't feed into a downstream supervised model the way PCA's often does.
• Never read cluster size or inter-cluster distance quantitatively off a t-SNE plot — only local neighbor relationships are reliably preserved.
• LDA is the one technique in this notebook that uses label information, making it fundamentally different in purpose from PCA and t-SNE, both of which are entirely unsupervised.

Best Practices Checklist#

• Always scale features before applying PCA
• Use a scree plot or cumulative explained variance to choose the number of PCA components, rather than an arbitrary number
• Validate whether PCA preprocessing actually improves downstream model performance via cross-validation, rather than assuming it always helps
• Use t-SNE (or UMAP) specifically for visualization, not as preprocessing for another model
• Try multiple perplexity values with t-SNE rather than trusting a single run
• Never interpret cluster size or distance between clusters quantitatively from a t-SNE plot
• Consider LDA over PCA when the downstream task is specifically classification and labels are available
• Remember PCA can transform new/unseen data after fitting; t-SNE generally cannot