Backpropagation: Complete Notes (Beginner to Advanced)

---

1. Derivatives#

A derivative measures how much a function's output changes in response to a small change in its input. It represents the rate of change, or geometrically, the slope of the function at a given point.

$f'(x) = \frac{df}{dx} = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$

Why derivatives matter for neural networks: training a network means adjusting weights and biases to reduce the loss. To know how to adjust a weight (increase it or decrease it, and by how much), you need to know how sensitive the loss is to that weight. That sensitivity is exactly what a derivative measures.

Without derivatives: you would have no mathematical way to know whether increasing or decreasing a specific weight would reduce the loss, or by how much. You would be reduced to random guessing or exhaustive trial-and-error search over weight values, which is computationally infeasible for networks with millions of parameters.

With derivatives: you get a precise, direction-aware signal for every single parameter, telling you exactly which way to move it to reduce the loss.

Some common derivative rules used in backpropagation:

Code example (numerical derivative approximation):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
def numerical_derivative(f, x, h=1e-7):
    return (f(x + h) - f(x - h)) / (2 * h)

def square(x):
    return x ** 2

# Actual derivative of x^2 at x=3 is 2*3 = 6
print("Numerical derivative:", numerical_derivative(square, 3))

---

2. Partial Derivatives#

A partial derivative measures how a function changes with respect to one specific input variable, while treating all other input variables as constant. This is essential in neural networks because the loss is a function of many parameters (potentially millions of weights and biases) simultaneously.

For a function L(w1, w2, b), the partial derivative with respect to w1 is written:

dL/dw1

This tells you how the loss L changes if you nudge w1 slightly, while w2 and b remain fixed.

Without partial derivatives: you would only be able to measure how the loss changes when all parameters move together, giving you no way to isolate the individual contribution and required adjustment of a single specific weight among millions.

With partial derivatives: you get an individual, isolated update signal for every single weight and bias in the network independently, which is exactly what is needed to update each parameter correctly during training.

Example: for L(w1, w2) = w1^2 + 3*w1*w2 + w2^2:

Mathematical Formulation
Copy
dL/dw1 = 2*w1 + 3*w2
dL/dw2 = 3*w1 + 2*w2

Code example (partial derivatives with PyTorch autograd):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch

w1 = torch.tensor(2.0, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)

L = w1**2 + 3*w1*w2 + w2**2
L.backward()

print("dL/dw1:", w1.grad.item())  # expected: 2*2 + 3*3 = 13
print("dL/dw2:", w2.grad.item())  # expected: 3*2 + 2*3 = 12

---

3. Chain Rule#

The chain rule is a calculus rule for computing the derivative of a composite function (a function made up of nested functions). If y = f(g(x)), the chain rule states:

dy/dx = dy/dg * dg/dx

In words: the overall rate of change is the product of the rates of change of each individual step in the chain.

Why this matters for neural networks: a neural network is literally a long chain of nested functions. For example, a 3-layer network's output can be written as:

output = f3( f2( f1(x) ) )

Each layer's output feeds into the next layer as input, forming a deeply nested composite function. To find how the loss depends on a weight deep inside this chain (say, in the first layer), you must multiply together the derivatives of every function the signal passed through, from the loss all the way back to that weight.

Without the chain rule: you would have no systematic way to compute how a change in an early layer's weight affects the final loss, since the effect passes through many intermediate transformations. You could not train any network with more than one layer.

With the chain rule: you can compute the exact gradient of the loss with respect to any parameter at any depth in the network, by multiplying the local derivatives along the path from that parameter to the final loss. This is precisely the mathematical mechanism that backpropagation implements.

Example: if z = w*x + b, a = sigmoid(z), and L = (a - y_true)^2, then to find dL/dw:

dL/dw = dL/da * da/dz * dz/dw

Mathematical Formulation
Copy
dL/da = 2 * (a - y_true)
da/dz = sigmoid(z) * (1 - sigmoid(z))
dz/dw = x

Multiplying these three together gives the full gradient of the loss with respect to w.

Code example (manually verifying the chain rule):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = 2.0
w = 0.5
b = 0.1
y_true = 1.0

z = w * x + b
a = sigmoid(z)
L = (a - y_true) ** 2

# Chain rule: dL/dw = dL/da * da/dz * dz/dw
dL_da = 2 * (a - y_true)
da_dz = sigmoid(z) * (1 - sigmoid(z))
dz_dw = x

dL_dw = dL_da * da_dz * dz_dw
print("dL/dw via chain rule:", dL_dw)

---

4. Computational Graph#

A computational graph represents every operation in the forward pass as a node connected by edges representing data flow. During forward propagation, values flow from input nodes to the output/loss node. During backpropagation, gradients flow in the reverse direction: from the loss node back to every parameter node, using the chain rule at every step.

Example computational graph for L = (sigmoid(w*x + b) - y_true)^2:

Architecture & Data Flow
Copy
Forward direction (left to right):
  x, w --> [multiply] --> wx --> [add b] --> z --> [sigmoid] --> a --> [subtract y_true, square] --> L

Backward direction (right to left, gradients flow this way):
  dL/dL=1 --> dL/da --> dL/dz --> dL/d(wx) --> dL/dw  and  dL/dx
                                             --> dL/db

At each node during the backward pass, the local gradient of that node's operation is multiplied by the incoming gradient from the node ahead of it (closer to the loss). This is the chain rule applied node-by-node, which is exactly how backpropagation is implemented computationally.

Without representing the network as a computational graph: frameworks would have no structured way to know which operations depend on which, making it impossible to systematically apply the chain rule across the entire network automatically.

With a computational graph: frameworks like PyTorch and TensorFlow can record every operation as it happens (this is called "tracing"), then automatically walk the graph backward, applying the chain rule at each node, to compute gradients for every parameter with a single .backward() call. This is called automatic differentiation (autograd).

Code example (PyTorch builds and traverses the graph automatically):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch

x = torch.tensor(2.0, requires_grad=True)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.1, requires_grad=True)
y_true = torch.tensor(1.0)

z = w * x + b
a = torch.sigmoid(z)
L = (a - y_true) ** 2

L.backward()  # traverses the computational graph backward

print("dL/dw:", w.grad.item())
print("dL/db:", b.grad.item())
print("dL/dx:", x.grad.item())

---

5. Gradient#

A gradient is a vector containing the partial derivative of a function with respect to every one of its input variables. For a loss function L that depends on many weights w1, w2, ..., wn, the gradient is:

gradient = [ dL/dw1, dL/dw2, ..., dL/dwn ]

Interpretation: the gradient points in the direction of the steepest increase of the function. Since we want to minimize the loss, we move in the opposite direction of the gradient. This is the core idea behind gradient descent:

w_new = w_old - learning_rate * dL/dw

Where learning_rate is a small positive number controlling the step size of each update.

Without the gradient: you would have no directional signal at all for a multi-parameter function; you'd only know local slopes for one variable at a time in isolation, and no coherent way to combine millions of such signals into a single coordinated update step across the whole network.

With the gradient: every parameter in the entire network receives a precise, simultaneous update signal, all derived consistently from the same single backward pass through the computational graph.

Code example (one step of gradient descent):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch

w = torch.tensor(0.5, requires_grad=True)
x = torch.tensor(2.0)
y_true = torch.tensor(1.0)
learning_rate = 0.1

# Forward pass
z = w * x
loss = (z - y_true) ** 2

# Backward pass (compute gradient)
loss.backward()
print("Gradient dL/dw:", w.grad.item())

# Manual gradient descent update
with torch.no_grad():
    w -= learning_rate * w.grad

print("Updated w:", w.item())

---

6. Backpropagation#

Backpropagation ("backward propagation of errors") is the algorithm that efficiently computes the gradient of the loss with respect to every weight and bias in the network, using the chain rule, by working backward from the output layer to the input layer.

High-level steps of backpropagation:

1. Forward pass: compute the output of the network and the resulting loss.
2. Compute the gradient of the loss with respect to the output (the last layer's activation).
3. Move backward layer by layer. At each layer, use the chain rule to compute:
   • The gradient of the loss with respect to that layer's weights and bias (used to update them).
   • The gradient of the loss with respect to that layer's input (passed backward to the previous layer, to continue the chain).
4. Repeat until you reach the input layer.
5. Update all weights and biases using the computed gradients (typically via gradient descent or a variant like Adam).

Why backpropagation is efficient: without it, computing the gradient for each individual weight separately (by re-running the entire forward pass with a tiny change to that one weight, called the "numerical" or "finite difference" method) would require roughly as many forward passes as there are parameters, which is computationally infeasible for networks with millions or billions of parameters. Backpropagation computes gradients for all parameters in roughly the same time as a single forward pass, by reusing intermediate calculations through the chain rule.

Without backpropagation: training modern deep networks (which can have millions to billions of parameters) would be computationally impossible within any reasonable time budget.

With backpropagation: gradients for every parameter are computed efficiently in one backward pass, making it practical to train very large, very deep networks.

Code example (backpropagation from scratch for a single hidden layer network):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    s = sigmoid(z)
    return s * (1 - s)

# Simple network: input -> hidden layer (sigmoid) -> output layer (sigmoid) -> loss (MSE)
np.random.seed(0)
X = np.array([[0.5, 0.8]])       # 1 sample, 2 features
y_true = np.array([[1.0]])

W1 = np.random.randn(2, 3) * 0.1  # input(2) -> hidden(3)
b1 = np.zeros((1, 3))
W2 = np.random.randn(3, 1) * 0.1  # hidden(3) -> output(1)
b2 = np.zeros((1, 1))

# ---- FORWARD PASS ----
Z1 = np.dot(X, W1) + b1
A1 = sigmoid(Z1)
Z2 = np.dot(A1, W2) + b2
A2 = sigmoid(Z2)          # final prediction

loss = np.mean((A2 - y_true) ** 2)
print("Loss:", loss)

# ---- BACKWARD PASS (backpropagation) ----
# Step 1: gradient of loss w.r.t. output activation A2
dL_dA2 = 2 * (A2 - y_true)

# Step 2: gradient w.r.t. Z2 (apply chain rule through sigmoid)
dL_dZ2 = dL_dA2 * sigmoid_derivative(Z2)

# Step 3: gradients for output layer's weights and bias
dL_dW2 = np.dot(A1.T, dL_dZ2)
dL_db2 = np.sum(dL_dZ2, axis=0, keepdims=True)

# Step 4: propagate gradient back to hidden layer's activation
dL_dA1 = np.dot(dL_dZ2, W2.T)

# Step 5: gradient w.r.t. Z1 (apply chain rule through sigmoid)
dL_dZ1 = dL_dA1 * sigmoid_derivative(Z1)

# Step 6: gradients for hidden layer's weights and bias
dL_dW1 = np.dot(X.T, dL_dZ1)
dL_db1 = np.sum(dL_dZ1, axis=0, keepdims=True)

print("Gradient dL/dW2:\n", dL_dW2)
print("Gradient dL/dW1:\n", dL_dW1)

# ---- UPDATE WEIGHTS (gradient descent step) ----
learning_rate = 0.1
W2 -= learning_rate * dL_dW2
b2 -= learning_rate * dL_db2
W1 -= learning_rate * dL_dW1
b1 -= learning_rate * dL_db1

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# The same network trained with PyTorch's automatic backpropagation
import torch
import torch.nn as nn

torch.manual_seed(0)

model = nn.Sequential(
    nn.Linear(2, 3),
    nn.Sigmoid(),
    nn.Linear(3, 1),
    nn.Sigmoid()
)

X = torch.tensor([[0.5, 0.8]])
y_true = torch.tensor([[1.0]])

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# Forward pass
y_pred = model(X)
loss = loss_fn(y_pred, y_true)
print("Loss:", loss.item())

# Backward pass (backpropagation happens automatically here)
optimizer.zero_grad()
loss.backward()

# Update weights
optimizer.step()

---

7. Gradient Flow#

Gradient flow refers to how gradient values move backward through the layers of a network during backpropagation, and specifically, how their magnitude changes as they pass through each layer.

Since each layer's gradient computation involves multiplying by the local derivative of that layer's activation function and its weights, gradients are essentially the product of many numbers chained together across all the layers between the loss and a given parameter:

dL/dw(layer 1) = dL/dA(last layer) * (product of many local derivatives across every layer in between) * dz/dw(layer 1)

Healthy gradient flow means these gradients stay in a reasonable numerical range (neither vanishing to near-zero nor exploding to very large values) as they travel backward through many layers, allowing every layer, including early ones, to keep learning effectively.

Without monitoring/managing gradient flow: deep networks can silently fail to train properly. Early layers may receive gradients so small that their weights barely change (they stop learning), or so large that training becomes unstable and diverges.

With good gradient flow (achieved through proper weight initialization, normalization techniques like Batch Normalization, activation function choice like ReLU, and architectural aids like residual/skip connections): gradients remain stable in magnitude across many layers, enabling even very deep networks (hundreds of layers) to train effectively.

This concept directly leads into the two major, well-known gradient flow problems below.

---

8. Vanishing Gradients#

The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through many layers, effectively shrinking toward zero by the time they reach the earlier layers of the network.

Root cause: Because of the chain rule, the gradient for an early layer is the product of many local derivatives from all the layers between it and the loss. If most of these local derivatives are small fractions (less than 1), multiplying many of them together makes the result exponentially smaller as depth increases.

This is especially severe with sigmoid and tanh activation functions:

• The maximum value of the sigmoid derivative is only 0.25 (occurring at z=0), and it approaches 0 for large positive or negative z.
• The maximum value of the tanh derivative is 1 (at z=0), but it also approaches 0 for large |z|.

If a network has, say, 10 layers all using sigmoid, and the local derivative at each layer averages around 0.2, the combined gradient for the very first layer would be roughly 0.2^10 ≈ 0.0000001024, an extremely tiny number. This means the first layer's weights receive almost no meaningful update signal, so it essentially stops learning.

Without addressing vanishing gradients: deep networks using sigmoid/tanh activations throughout become nearly impossible to train past a handful of layers, since the earliest layers never receive a usable learning signal, effectively "starving" them of updates while later layers may still learn somewhat normally.

With solutions to vanishing gradients, deep networks can be trained successfully. Common solutions include:

• Using ReLU (or its variants like Leaky ReLU/ELU/GELU) instead of sigmoid/tanh in hidden layers, since ReLU's derivative is exactly 1 for all positive inputs, which does not shrink with depth.
• Proper weight initialization (e.g., He initialization for ReLU networks, Xavier/Glorot initialization for sigmoid/tanh networks), which keeps the variance of activations and gradients stable across layers.
• Batch Normalization, which normalizes layer inputs to keep them in a well-behaved numerical range, preventing activations (and thus gradients) from saturating.
• Residual/skip connections (as used in ResNet architectures), which provide a direct path for gradients to flow backward without being forced through every single intermediate layer's multiplication, effectively shortening the multiplicative chain.

Code example (demonstrating vanishing gradients with sigmoid in a deep network):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

torch.manual_seed(0)

# A deep network using sigmoid activations throughout
class DeepSigmoidNet(nn.Module):
    def __init__(self, num_layers=10):
        super().__init__()
        layers = []
        for _ in range(num_layers):
            layers.append(nn.Linear(10, 10))
            layers.append(nn.Sigmoid())
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)

model = DeepSigmoidNet(num_layers=10)
X = torch.randn(1, 10)
y_true = torch.randn(1, 10)

output = model(X)
loss = nn.MSELoss()(output, y_true)
loss.backward()

# Inspect gradient magnitude at the very first layer vs the very last layer
first_layer_grad = model.network[0].weight.grad.abs().mean().item()
last_layer_grad = model.network[-2].weight.grad.abs().mean().item()

print("Average gradient magnitude, first layer:", first_layer_grad)
print("Average gradient magnitude, last layer:", last_layer_grad)
# Typically, first_layer_grad will be dramatically smaller than last_layer_grad

---

9. Exploding Gradients#

The exploding gradient problem is the opposite of vanishing gradients: gradients become extremely large as they are propagated backward through many layers, growing exponentially instead of shrinking.

Root cause: if the local derivatives (or weight values) being multiplied together across layers are consistently greater than 1, the chain rule's repeated multiplication causes the combined gradient to grow exponentially with depth, rather than shrink. This is common when weights are initialized with large values, or in very deep networks and recurrent neural networks (RNNs) processing long sequences.

Symptoms of exploding gradients:

• Loss values suddenly become very large or turn into NaN (Not a Number) during training.
• Model weights update by huge amounts each step, causing training to become unstable and diverge instead of converge.

Without addressing exploding gradients: training can become numerically unstable, weights can be pushed to extreme values in a single update, and the model can fail to converge at all, sometimes crashing entirely due to numerical overflow.

With solutions to exploding gradients, training remains stable. Common solutions include:

• Gradient clipping, which caps the gradient's magnitude (or norm) at a maximum threshold before applying the weight update, preventing any single update from being too extreme.
• Proper weight initialization (same techniques as for vanishing gradients: He or Xavier initialization), which helps keep the multiplicative chain closer to a stable range from the start.
• Batch Normalization, which helps stabilize the scale of activations (and consequently gradients) throughout the network.
• Lower learning rates, which reduce the size of each weight update even if the raw gradient is large.
• Using architectures designed to handle long chains better, such as LSTM/GRU cells instead of plain RNNs for sequential data, which include internal gating mechanisms specifically designed to control gradient flow over long sequences.

Code example (gradient clipping in PyTorch):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)

X = torch.randn(1, 10)
y_true = torch.randn(1, 10)

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

output = model(X)
loss = loss_fn(output, y_true)

optimizer.zero_grad()
loss.backward()

# Clip gradients to a maximum norm of 1.0 before updating weights
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

optimizer.step()

Vanishing vs Exploding Gradients Comparison#

---

Quick Recap (Beginner to Advanced Flow)#

1. A derivative measures the rate of change of a function; a partial derivative does the same for one variable in a multi-variable function while holding others fixed.
2. The chain rule lets you compute the derivative of a nested (composite) function by multiplying the derivatives of each step, which is essential because a neural network is a long chain of nested layer functions.
3. The computational graph represents the network's operations as connected nodes, allowing the chain rule to be applied systematically and automatically (autograd) as gradients flow backward through it.
4. The gradient is the vector of all partial derivatives of the loss with respect to every parameter, pointing in the direction of steepest increase; moving opposite to it reduces the loss.
5. Backpropagation is the efficient algorithm that computes this gradient for every parameter in the network in one backward pass, by applying the chain rule layer by layer from the output back to the input.
6. Gradient flow describes how gradient magnitude changes as it moves backward through layers; keeping this flow healthy is critical for training deep networks.
7. Vanishing gradients occur when gradients shrink toward zero through many layers (common with sigmoid/tanh), stalling learning in early layers.
8. Exploding gradients occur when gradients grow exponentially through many layers, destabilizing training; solved primarily through gradient clipping and careful initialization.