Neural Networks: Multi-Layer Perceptron Basics

Overview#

This final notebook in the supervised algorithms section covers the Multi-Layer Perceptron (MLP) — the foundational neural network architecture that everything in deep learning ultimately builds on. It goes deep into forward propagation, activation functions and why non-linearity is essential, backpropagation (the algorithm that actually trains the network), and a thorough, honest comparison of when a neural network is worth the added complexity over the classical algorithms covered earlier in this section.

1. From a Single Neuron to a Network#

A single artificial neuron performs the exact same computation as logistic regression: a weighted sum of inputs, passed through an activation function.

Python
CopyAsk AI
import numpy as np

def single_neuron(x, w, b, activation_fn):
    z = np.dot(w, x) + b
    return activation_fn(z)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = np.array([1.0, 2.0, 3.0])
w = np.array([0.5, -0.3, 0.8])
b = 0.1

output = single_neuron(x, w, b, sigmoid)
print(f"Single neuron output: {output:.4f}")

A single neuron, on its own, can only represent a linear decision boundary — exactly as limited as logistic regression. The power of neural networks comes from stacking many neurons into layers, and stacking multiple layers on top of each other.

2. The MLP Architecture#

text
Ask AICopy
Input layer: one node per feature — no computation happens here, it just
              passes the raw feature values forward

Hidden layer(s): each neuron computes a weighted sum of ALL outputs from
                   the previous layer, adds a bias, and applies an
                   activation function — this is where the actual
                   "learning" of representations happens

Output layer: produces the final prediction — a single neuron
                (regression or binary classification) or multiple neurons
                (multiclass classification, one per class)

Python
CopyAsk AI
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

mlp = MLPClassifier(hidden_layer_sizes=(16, 8), activation="relu", max_iter=1000, random_state=42)
mlp.fit(X_train_scaled, y_train)

print(f"Test accuracy: {mlp.score(X_test_scaled, y_test):.3f}")

hidden_layer_sizes=(16, 8) means two hidden layers: the first with 16 neurons, the second with 8 — this crescent-moon-shaped, non-linearly separable data (the same make_moons dataset that broke K-Means in the Clustering notebook) is exactly the kind of problem where an MLP's non-linear capability genuinely matters, unlike plain logistic regression.

3. Forward Propagation, Step by Step#

Python
CopyAsk AI
import numpy as np

def relu(z):
    return np.maximum(0, z)

def forward_pass_manual(x, W1, b1, W2, b2):
    """A 2-layer network: 1 hidden layer, 1 output layer."""
    # Layer 1 (hidden)
    z1 = W1 @ x + b1        # weighted sum for every neuron in the hidden layer
    a1 = relu(z1)              # activation applied elementwise

    # Layer 2 (output)
    z2 = W2 @ a1 + b2
    a2 = sigmoid(z2)             # sigmoid for binary classification output

    return a2

# Example: 3 input features, 4 hidden neurons, 1 output
np.random.seed(42)
x = np.array([1.0, -0.5, 2.0])
W1 = np.random.randn(4, 3) * 0.1   # (hidden_size, input_size)
b1 = np.zeros(4)
W2 = np.random.randn(1, 4) * 0.1    # (output_size, hidden_size)
b2 = np.zeros(1)

output = forward_pass_manual(x, W1, b1, W2, b2)
print(f"Forward pass output: {output}")

Forward propagation is simply this sequence of matrix multiplications and activation functions, applied layer by layer — conceptually straightforward; the real complexity of neural networks lies entirely in how the weights are learned (backpropagation, next).

4. Why Non-Linear Activation Functions Are Essential#

Python
CopyAsk AI
# What happens WITHOUT a non-linear activation function between layers?
def linear_activation(z):
    return z   # no transformation at all

def forward_pass_no_nonlinearity(x, W1, b1, W2, b2):
    z1 = W1 @ x + b1
    a1 = linear_activation(z1)   # NO non-linearity
    z2 = W2 @ a1 + b2
    return z2

# Mathematically, this entire 2-layer computation collapses to:
#   output = W2 @ (W1 @ x + b1) + b2
#          = (W2 @ W1) @ x + (W2 @ b1 + b2)
#          = W_combined @ x + b_combined
#
# ...which is EXACTLY equivalent to a SINGLE linear layer, no matter how
# many layers you stack, as long as none of them have a non-linear
# activation function.

text
Ask AICopy
This is the single most important fact about why deep networks work at
all: WITHOUT non-linear activation functions, stacking any number of
layers is mathematically equivalent to a single linear transformation —
all the representational power of "depth" would be completely lost.
Non-linear activations are what allow a network to represent genuinely
non-linear, complex decision boundaries and functions.

Common activation functions, compared#

Python
CopyAsk AI
import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z)
def relu(z): return np.maximum(0, z)
def leaky_relu(z, alpha=0.01): return np.where(z > 0, z, alpha * z)

z_values = np.linspace(-5, 5, 100)
# Plotting each of these shows their characteristic shapes

Architecture & Data Flow
Copy
Sigmoid: output range (0, 1). Historically popular, but suffers badly
  from the VANISHING GRADIENT problem (see section 6) — its derivative
  is very small for large |z|, which slows or stalls learning in deep
  networks. Still used at OUTPUT layers for binary classification.

Tanh: output range (-1, 1), zero-centered (unlike sigmoid). Similar
  vanishing gradient issue, though somewhat less severe than sigmoid due
  to being zero-centered.

ReLU (Rectified Linear Unit): output = max(0, z). The modern DEFAULT for
  hidden layers — computationally cheap, and its derivative is either
  exactly 0 or exactly 1, largely avoiding vanishing gradients for
  positive inputs. Downside: "dying ReLU" — a neuron can get stuck
  always outputting 0 if its weights end up in a state where z is always
  negative, at which point its gradient is also always 0 and it can
  never recover through gradient descent.

Leaky ReLU: like ReLU, but allows a small, non-zero gradient (alpha * z)
  for negative inputs, specifically to address the dying ReLU problem.

5. Backpropagation: How the Network Actually Learns#

Backpropagation computes the gradient of the loss function with respect to EVERY weight in the network, using the chain rule from calculus, working backward from the output layer to the input layer.

text
Ask AICopy
The chain rule, in essence:
  If loss depends on the output, and the output depends on a hidden
  layer's weights, then:

  d(loss)/d(hidden_weight) = d(loss)/d(output) * d(output)/d(hidden_layer)
                              * d(hidden_layer)/d(hidden_weight)

Backpropagation computes this efficiently by working BACKWARD: first
compute d(loss)/d(output) at the output layer, then use it to compute
the gradient for the LAST layer's weights, then propagate that
gradient information backward to compute the SECOND-TO-LAST layer's
weight gradients, and so on, all the way back to the input.

Python
CopyAsk AI
# A simplified, manual backpropagation step for a tiny 1-hidden-layer
# network, illustrating the mechanism (not a production implementation)

def backprop_step(x, y_true, W1, b1, W2, b2, learning_rate=0.1):
    # Forward pass
    z1 = W1 @ x + b1
    a1 = relu(z1)
    z2 = W2 @ a1 + b2
    y_pred = sigmoid(z2)

    # Backward pass: compute gradients, starting from the OUTPUT
    d_loss_d_z2 = y_pred - y_true                   # derivative of log loss + sigmoid, combined (a clean simplification)
    d_loss_d_W2 = np.outer(d_loss_d_z2, a1)            # gradient for output layer weights
    d_loss_d_b2 = d_loss_d_z2

    d_loss_d_a1 = W2.T @ d_loss_d_z2                     # propagate the gradient BACKWARD into the hidden layer
    d_loss_d_z1 = d_loss_d_a1 * (z1 > 0)                    # ReLU derivative: 1 where z1>0, else 0
    d_loss_d_W1 = np.outer(d_loss_d_z1, x)                    # gradient for hidden layer weights
    d_loss_d_b1 = d_loss_d_z1

    # Update weights: take a small step in the direction that REDUCES loss
    W2 -= learning_rate * d_loss_d_W2
    b2 -= learning_rate * d_loss_d_b2
    W1 -= learning_rate * d_loss_d_W1
    b1 -= learning_rate * d_loss_d_b1

    return W1, b1, W2, b2

This is repeated for every training example (or batch of examples), across many passes through the full dataset (epochs), gradually adjusting every weight in the network to reduce the overall loss — this entire process is what scikit-learn's MLPClassifier.fit(), and every deep learning framework, is doing internally, just far more efficiently and at much larger scale.

6. The Vanishing Gradient Problem#

Architecture & Data Flow
Copy
Notice in the backprop derivation above: the gradient flowing backward
gets MULTIPLIED by the activation function's derivative at each layer it
passes through. For sigmoid and tanh, this derivative is ALWAYS less
than 1 (often much less, for large |z|).

In a DEEP network (many layers), the gradient gets multiplied by many
such small numbers in sequence — and the product of many numbers less
than 1 shrinks toward zero EXPONENTIALLY with depth.

The practical consequence: gradients reaching the EARLIEST layers of a
deep sigmoid/tanh network become vanishingly small, meaning those layers'
weights barely update at all — the network effectively stops learning in
its earlier layers, even though later layers continue training normally.

This is precisely why ReLU became the standard choice for hidden layers in modern networks — its derivative doesn't shrink toward zero for positive inputs, allowing gradients to flow backward through many layers without vanishing nearly as severely.

7. Key Hyperparameters#

Python
CopyAsk AI
from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(
    hidden_layer_sizes=(64, 32),    # network architecture — more/larger layers = more capacity, more overfitting risk
    activation="relu",                 # activation function for hidden layers
    solver="adam",                       # optimization algorithm (adam is the modern default, adaptive learning rate)
    alpha=0.0001,                          # L2 regularization strength — directly analogous to Ridge's alpha
    learning_rate_init=0.001,                # initial step size for gradient descent updates
    batch_size="auto",                         # number of samples processed before each weight update
    max_iter=500,                                # maximum number of epochs
    early_stopping=True,                           # stop training once validation score stops improving
    validation_fraction=0.1,                         # portion of training data held out for early stopping checks
    random_state=42,
)

text
Ask AICopy
alpha (L2 regularization) works IDENTICALLY in spirit to Ridge
regression's alpha, covered in the Regularization Techniques notebook —
it penalizes large weight magnitudes, directly controlling the
bias-variance tradeoff for the network as a whole.

The effect of network size on overfitting#

Python
CopyAsk AI
from sklearn.model_selection import cross_val_score

for hidden_sizes in [(4,), (16,), (64,), (128, 64)]:
    mlp = MLPClassifier(hidden_layer_sizes=hidden_sizes, max_iter=1000, random_state=42)
    scores = cross_val_score(mlp, X_train_scaled, y_train, cv=5)
    print(f"{hidden_sizes}: CV accuracy = {scores.mean():.3f} (+/- {scores.std():.3f})")

Larger, deeper networks have more parameters and therefore more capacity to fit complex patterns — but also more capacity to overfit, especially on smaller datasets, exactly matching the bias-variance/complexity principles established throughout this entire course, just with "network size" as the new complexity dial.

8. Feature Scaling: Even More Critical Than for Other Algorithms#

Python
CopyAsk AI
# Neural networks are ESPECIALLY sensitive to feature scale, more so
# than most classical algorithms — because gradient descent's step size
# (learning_rate) interacts directly with the SCALE of the inputs and
# gradients; unscaled features can cause training to converge extremely
# slowly, or fail to converge at all

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

mlp_pipeline = make_pipeline(StandardScaler(), MLPClassifier(hidden_layer_sizes=(32,), max_iter=1000))

Always scale features before training an MLP — this is one of the least optional preprocessing steps in this entire course, given how directly gradient-based optimization is affected by input scale.

9. When Neural Networks Are (and Aren't) Worth It#

text
Ask AICopy
Neural networks tend to be worth the added complexity when:
- The dataset is LARGE (the classical algorithms in this section often
  match or beat small MLPs on small-to-moderate tabular datasets)
- The problem involves unstructured data (images, audio, text) —
  though CNNs, transformers, and other specialized architectures beyond
  the scope of this notebook are typically used there, not plain MLPs
- Highly complex, non-linear feature interactions exist that would be
  impractical to engineer manually for a classical algorithm
- There's sufficient compute and tuning budget — networks generally
  need more careful hyperparameter tuning than tree ensembles

Neural networks are usually NOT the first choice when:
- The dataset is small-to-moderate and TABULAR/structured — gradient
  boosting (XGBoost/LightGBM/CatBoost, covered earlier in this section)
  frequently matches or outperforms neural networks on this exact data
  type, while requiring far less tuning and training time
- Interpretability is a hard requirement — MLPs are essentially
  "black boxes," even more so than Random Forest or Gradient Boosting
- Training time/compute resources are limited

Notes#

• A single neuron is mathematically equivalent to logistic regression — the power of neural networks comes entirely from stacking many neurons across multiple layers.
• Without non-linear activation functions, any number of stacked layers collapses mathematically to a single linear transformation — this is the precise reason non-linearity is essential, not just a design preference.
• Backpropagation is simply the chain rule from calculus, applied systematically backward through the network to compute every weight's gradient efficiently.
• The vanishing gradient problem, caused by repeatedly multiplying small derivative values through many layers, is why ReLU replaced sigmoid/tanh as the standard hidden-layer activation in modern networks.
• For small-to-moderate tabular datasets specifically, gradient boosting frequently outperforms neural networks while requiring substantially less tuning — MLPs are not automatically the "best" choice just because they're more sophisticated.

Best Practices Checklist#

• Always scale features before training an MLP — this matters even more than for most classical algorithms
• Use ReLU (or a variant like Leaky ReLU) for hidden layers by default, reserving sigmoid/tanh mainly for specific output layer needs
• Use early_stopping=True to prevent overtraining, rather than guessing a fixed max_iter
• Tune alpha (L2 regularization) and network size together, understanding both control the same underlying bias-variance tradeoff
• Compare against gradient boosting as a baseline on tabular data before assuming a neural network is the better choice
• Expect neural networks to need more hyperparameter tuning and training data than tree ensembles to reach comparable tabular-data performance
• Treat network architecture (layer count and sizes) as a genuine hyperparameter to validate via cross-validation, not a fixed default
• Reserve neural networks specifically for problems involving unstructured data or very large datasets, where their advantages are most pronounced