Forward Propagation: Complete Notes (Beginner to Advanced)

---

1. Forward Propagation#

Forward propagation (also called the forward pass) is the process of passing input data through a neural network, layer by layer, to produce an output. It is the mechanism by which a neural network makes a prediction.

The overall flow:

Input -> Linear Transformation -> Activation -> (repeat for each layer) -> Output

At each layer, two operations happen in sequence:

1. A linear transformation (weighted sum of inputs plus bias).
2. A non-linear activation function applied to that result.

This repeats layer after layer until the final output layer produces the prediction.

Without forward propagation: there is no defined way to compute an output from the input data at all. You cannot generate a prediction, and consequently you cannot compute a loss or train the network, since training requires comparing a predicted output against the true target.

With forward propagation: the network has a clear, deterministic (given fixed weights) path from raw input to prediction, which is the first of the two core steps in training a neural network (the second being backpropagation, which computes gradients to update the weights).

Important distinction: forward propagation happens the same way during both training and inference (prediction on new data). The only difference is that during training, the output of the forward pass is used to compute a loss, which then drives backpropagation; during inference, the output is simply used as the final prediction.

---

2. Linear Layer#

A linear layer (also called a dense layer or fully connected layer) is where the linear transformation of forward propagation happens. Every neuron in a linear layer is connected to every neuron in the previous layer.

Formula for a single layer:

Z = X . W + b

Where:

• X = input matrix, shape (batch_size, input_features)
• W = weight matrix, shape (input_features, output_neurons)
• b = bias vector, shape (output_neurons,)
• Z = pre-activation output, shape (batch_size, output_neurons)

After computing Z, an activation function f is applied element-wise:

A = f(Z)

A is called the activation of that layer, and it becomes the input X for the next layer.

Without the linear layer's matrix formulation: you would have to compute each neuron's weighted sum individually in a loop, which is both slower and harder to express in vectorized code that GPUs can accelerate.

With the linear layer as a matrix operation: the entire layer's computation for an entire batch of samples can be done in a single matrix multiplication, which is exactly how deep learning frameworks (PyTorch, TensorFlow) achieve speed on GPUs.

Code example (a linear layer from scratch with NumPy):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def linear_layer(X, W, b):
    return np.dot(X, W) + b

# 2 samples, 3 features each
X = np.array([[1.0, 2.0, 3.0],
              [0.5, 0.1, 0.9]])

# 3 inputs -> 4 neurons
W = np.random.randn(3, 4) * 0.01
b = np.zeros(4)

Z = linear_layer(X, W, b)
print("Pre-activation output Z:\n", Z)

Code example (a linear layer using PyTorch):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

linear = nn.Linear(in_features=3, out_features=4)

X = torch.tensor([[1.0, 2.0, 3.0],
                   [0.5, 0.1, 0.9]])

Z = linear(X)
print("Pre-activation output Z:\n", Z)

---

3. Logits#

Logits are the raw, unnormalized output scores produced by the final linear layer of a network, before any final activation function (like softmax or sigmoid) is applied.

• Logits can be any real number: positive, negative, small, or large.
• They represent relative confidence across classes but are not probabilities themselves.
• Example: logits of [2.5, -1.0, 0.3] mean the first class has the highest raw score, but these numbers do not sum to 1 and cannot be directly interpreted as percentages.

Without treating raw scores as logits (i.e., trying to interpret them directly as probabilities): you would misinterpret the model's confidence, since logits are not bounded between 0 and 1 and don't sum to 1 across classes.

With the concept of logits clearly separated from probabilities: you know that a further transformation (softmax for multi-class, sigmoid for binary) is required to convert these raw scores into interpretable probabilities.

Why frameworks often keep logits separate: for numerical stability, loss functions like CrossEntropyLoss in PyTorch or from_logits=True in TensorFlow/Keras expect raw logits rather than post-softmax probabilities, since combining the softmax and the loss calculation internally is more numerically stable than doing them as two separate steps.

Code example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

logits = torch.tensor([[2.5, -1.0, 0.3]])
print("Raw logits:", logits)

# PyTorch's CrossEntropyLoss expects raw logits, not softmax probabilities
target = torch.tensor([0])  # true class index
loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(logits, target)
print("Loss computed directly from logits:", loss.item())

Without passing raw logits to CrossEntropyLoss (and instead pre-applying softmax yourself): you risk applying softmax twice internally, which distorts the gradient computation and can cause numerically incorrect or unstable training. With passing raw logits directly: the loss function applies the numerically stable combined operation (log-softmax + negative log-likelihood) internally.

---

4. Output Probabilities#

Output probabilities are the final, normalized values produced by applying an appropriate activation function to the logits. They represent the model's confidence for each possible outcome, and they must satisfy the mathematical rules of a probability distribution.

For multi-class classification (Softmax):

probability_i = e^(logit_i) / sum(e^(logit_j) for all j)

• All output probabilities sum to exactly 1.
• Each individual probability is between 0 and 1.

For binary classification (Sigmoid):

probability = 1 / (1 + e^(-logit))

• A single probability between 0 and 1, representing the likelihood of the positive class. The negative class probability is simply 1 - probability.

Without converting logits to probabilities: you cannot make a meaningful, interpretable decision from the model's output. You also cannot properly compute standard loss functions like cross-entropy that are mathematically defined in terms of probabilities.

With output probabilities: you can interpret the model's confidence directly (e.g., "87% confident this is a cat"), and you can apply a decision threshold (like 0.5 for binary classification) to make a final class prediction.

Code example:

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def softmax(logits):
    exp_logits = np.exp(logits - np.max(logits))  # numerical stability
    return exp_logits / np.sum(exp_logits)

def sigmoid(logit):
    return 1 / (1 + np.exp(-logit))

# Multi-class example
logits = np.array([2.5, -1.0, 0.3])
probs = softmax(logits)
print("Multi-class probabilities:", probs, "Sum:", np.sum(probs))

# Binary example
binary_logit = 1.2
binary_prob = sigmoid(binary_logit)
print("Binary probability:", binary_prob)

---

5. Computational Graph#

A computational graph is a way of representing a series of mathematical operations as a directed graph, where:

• Nodes represent operations (addition, multiplication, matrix multiplication, activation functions, etc.) or variables (inputs, weights, biases).
• Edges represent the flow of data (tensors) between operations.

Forward propagation is essentially the process of traversing this graph from input nodes to output nodes, computing each operation's result along the way.

Example computational graph for a single neuron output = sigmoid(w*x + b):

Architecture & Data Flow
Copy
x ---\
      * ---> (w*x) ---\
w ---/                 + ---> z ---> sigmoid ---> output
                       /
b --------------------/

Why computational graphs matter beyond forward propagation:

Deep learning frameworks like PyTorch and TensorFlow build this graph automatically as you perform operations. This graph is not just for computing the forward pass; it is also what makes automatic differentiation (autograd) possible. Each node in the graph "remembers" how it was computed, so during backpropagation, the framework can walk the graph backward and apply the chain rule automatically to compute gradients for every weight and bias.

Without a computational graph: you would have to manually derive and code the gradient (derivative) calculations for every single operation in your network by hand, which becomes impractical for anything beyond the simplest models.

With a computational graph and automatic differentiation: frameworks can compute exact gradients for arbitrarily complex networks automatically, which is the foundation that makes training deep networks with backpropagation practical.

Code example (PyTorch automatically builds the computational graph):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch

x = torch.tensor(2.0, requires_grad=True)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)

z = w * x + b              # linear transformation
output = torch.sigmoid(z)  # activation

print("Forward pass output:", output.item())

# PyTorch has built a computational graph in the background.
# Calling backward() traverses this graph in reverse to compute gradients.
output.backward()

print("Gradient of output with respect to w:", w.grad.item())
print("Gradient of output with respect to x:", x.grad.item())
print("Gradient of output with respect to b:", b.grad.item())

---

6. Multi-Layer Forward Pass#

A multi-layer forward pass extends the single-layer forward propagation concept across an entire network with multiple hidden layers. The output (activation) of one layer becomes the input to the next layer, and this repeats until the final output layer is reached.

General formula for layer l:

Mathematical Formulation
Copy
Z[l] = A[l-1] . W[l] + b[l]
A[l] = f(Z[l])

Where:

• A[0] = X (the original input to the network)
• W[l] and b[l] are the weights and bias for layer l
• f is the activation function for that layer (often ReLU for hidden layers, and softmax/sigmoid/linear for the output layer depending on the task)
• A[l] is the activation output of layer l, which feeds into layer l+1

Step-by-step for a 3-layer network (2 hidden layers + 1 output layer):

Mathematical Formulation
Copy
A[0] = X                              (input)
Z[1] = A[0] . W[1] + b[1]
A[1] = ReLU(Z[1])                     (hidden layer 1 output)

Z[2] = A[1] . W[2] + b[2]
A[2] = ReLU(Z[2])                     (hidden layer 2 output)

Z[3] = A[2] . W[3] + b[3]
A[3] = Softmax(Z[3])                  (final output probabilities)

Without correctly chaining each layer's output as the next layer's input: the network's layers would be computing independently on the same original input rather than progressively building more abstract representations, which defeats the entire purpose of depth in a deep neural network.

With a properly chained multi-layer forward pass: each layer learns to extract increasingly abstract features from the output of the previous layer (for example, in image models: edges in early layers, shapes in middle layers, and object parts in later layers).

Code example (multi-layer forward pass from scratch with NumPy):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np

def relu(z):
    return np.maximum(0, z)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

def forward_pass(X, params):
    A = X
    L = len(params) // 2  # number of layers (each layer has W and b)

    # Hidden layers with ReLU
    for l in range(1, L):
        Z = np.dot(A, params[f"W{l}"]) + params[f"b{l}"]
        A = relu(Z)

    # Output layer with Softmax
    Z_out = np.dot(A, params[f"W{L}"]) + params[f"b{L}"]
    output = softmax(Z_out)
    return output

# Example: 2 hidden layers, input size 4, hidden sizes 5 and 3, output size 2
np.random.seed(42)
params = {
    "W1": np.random.randn(4, 5) * 0.1, "b1": np.zeros(5),
    "W2": np.random.randn(5, 3) * 0.1, "b2": np.zeros(3),
    "W3": np.random.randn(3, 2) * 0.1, "b3": np.zeros(2),
}

X = np.array([[1.0, 0.5, -1.2, 0.3]])  # 1 sample, 4 features
output = forward_pass(X, params)
print("Final output probabilities:", output)

Code example (same multi-layer forward pass using PyTorch):

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import torch
import torch.nn as nn

class SimpleNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(4, 5)
        self.layer2 = nn.Linear(5, 3)
        self.output_layer = nn.Linear(3, 2)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        logits = self.output_layer(x)
        output = self.softmax(logits)
        return output

model = SimpleNetwork()
X = torch.tensor([[1.0, 0.5, -1.2, 0.3]])
output = model(X)
print("Final output probabilities:", output)

---

Summary: Full Forward Propagation Flow#

Architecture & Data Flow
Copy
Input (X)
   |
   v
Linear Layer 1: Z1 = X.W1 + b1
   |
   v
Activation (e.g., ReLU): A1 = ReLU(Z1)
   |
   v
Linear Layer 2: Z2 = A1.W2 + b2
   |
   v
Activation (e.g., ReLU): A2 = ReLU(Z2)
   |
   v
Output Linear Layer: Z_out = A2.W_out + b_out   --> these are the LOGITS
   |
   v
Final Activation (Softmax/Sigmoid): OUTPUT PROBABILITIES

This entire chain of operations forms a computational graph, which is traversed forward to get predictions and traversed backward (via backpropagation, covered separately) to compute gradients for training.

Quick Recap (Beginner to Advanced Flow)#

1. Forward propagation is the process of passing input through the network to produce an output.
2. Each linear layer performs Z = X.W + b, transforming the input into a new representation.
3. The final linear layer's raw, unnormalized outputs are called logits.
4. Logits are converted into output probabilities using softmax (multi-class) or sigmoid (binary).
5. All of these operations together form a computational graph, which enables automatic gradient computation during training.
6. A multi-layer forward pass chains this linear-plus-activation process across every layer, with each layer's output feeding into the next, allowing the network to build increasingly abstract representations of the data.