Intermediate

25 min read

#Feedforward#MLP#DNN#Universal Approximation#Architecture Design#PyTorch

10. Deep Feedforward Networks & MLP Architectures

Deep Feedforward Networks (MLP): universal approximation theorem, width vs depth tradeoffs, residual connections, and production network design.

Deep Feedforward Networks: Complete Notes (Beginner to Advanced)

Introduction#

A Deep Feedforward Network is a neural network in which information moves in a forward direction from the input toward the output without forming a recurrent loop.

The network is built by stacking layers of learnable transformations. In a basic feedforward architecture, the output of one layer becomes the input to the next layer.

Architecture & Data Flow
Input
  |
  v
Layer 1
  |
  v
Layer 2
  |
  v
Layer 3
  |
  v
Output

The main ideas that determine the structure and expressive power of a deep feedforward network are:

Multilayer Perceptron (MLP) — the standard fully connected feedforward architecture.
Deep Neural Network (DNN) — a network with multiple layers of computation.
Network Depth — how many sequential layers the network contains.
Network Width — how many neurons are present in a layer.
Parameter Count — how many learnable weights and biases the network contains.
Capacity — how complex a function the network can represent.
Skip Connections — connections that bypass one or more layers.
Residual Connections — a specific type of skip connection that adds the original input to a transformed version of that input.

1. Multilayer Perceptron (MLP)#

A Multilayer Perceptron (MLP) is a feedforward neural network made primarily from fully connected (dense) layers.

The term "perceptron" historically refers to a simple neuron/classifier. An MLP extends this idea by arranging neurons into multiple layers.

A typical MLP contains:

Architecture & Data Flow
Input Layer
     |
     v
Hidden Layer 1
     |
     v
Hidden Layer 2
     |
     v
Output Layer

Every neuron in one dense layer is connected to every neuron in the previous layer.

1.1 Basic MLP Structure#

Consider an MLP with:

4 input features
5 neurons in the first hidden layer
3 neurons in the second hidden layer
2 output neurons

text
Input          Hidden 1        Hidden 2        Output

x1  ───────┐
x2  ───────┼──> [ 5 neurons ] ──> [ 3 neurons ] ──> [ 2 neurons ]
x3  ───────┤
x4  ───────┘

Each layer performs a transformation:

Mathematical Formulation
Z = XW + b

For a hidden layer, a nonlinear activation is generally applied:

Mathematical Formulation
A = f(Z)

Therefore, a typical MLP can be represented as:

Mathematical Formulation
A1 = f(XW1 + b1)

A2 = f(A1W2 + b2)

Output = g(A2W3 + b3)

Here:

X = input
W = weights
b = bias
f = hidden-layer activation function
g = output-layer transformation/activation when required

1.2 Why Multiple Layers Are Used#

A single dense transformation can learn only a relatively simple transformation of its input.

By stacking layers, the network can repeatedly transform the representation:

Architecture & Data Flow
Raw Features
     |
     v
First Representation
     |
     v
Higher-Level Representation
     |
     v
More Complex Representation
     |
     v
Output

Each layer receives the representation produced by the previous layer.

Without multiple layers: the network has limited ability to build hierarchical transformations.

With multiple layers: the network can compose several transformations and represent substantially more complex functions.

1.3 Nonlinearity in an MLP#

The nonlinear activation between layers is important.

Suppose two layers contain only linear transformations:

Mathematical Formulation
Z1 = XW1 + b1

Z2 = Z1W2 + b2

The composition can still be represented as a single linear transformation:

Mathematical Formulation
Z2 = X(W1W2) + (b1W2 + b2)

Therefore, simply stacking linear layers does not provide the full expressive benefit of depth.

With a nonlinear activation:

Mathematical Formulation
Z1 = XW1 + b1
A1 = f(Z1)

Z2 = A1W2 + b2
A2 = f(Z2)

the network can represent nonlinear relationships.

1.4 MLP Using NumPy#


🐍 PythonInteractive WebAssembly
import numpy as np

def relu(x):
    return np.maximum(0, x)

# Input: 2 samples, 4 features
X = np.array([
    [1.0, 0.5, -1.0, 2.0],
    [0.2, 1.5,  0.3, 0.7]
])

# 4 -> 5 -> 3 -> 2
W1 = np.random.randn(4, 5) * 0.1
b1 = np.zeros(5)

W2 = np.random.randn(5, 3) * 0.1
b2 = np.zeros(3)

W3 = np.random.randn(3, 2) * 0.1
b3 = np.zeros(2)

A1 = relu(X @ W1 + b1)
A2 = relu(A1 @ W2 + b2)
output = A2 @ W3 + b3

print("Output shape:", output.shape)
print("Output:\n", output)

The dimensions flow as:

text
X       : (2, 4)
A1      : (2, 5)
A2      : (2, 3)
Output  : (2, 2)

1.5 MLP Using PyTorch#


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(4, 5),
    nn.ReLU(),
    nn.Linear(5, 3),
    nn.ReLU(),
    nn.Linear(3, 2)
)

X = torch.tensor([
    [1.0, 0.5, -1.0, 2.0],
    [0.2, 1.5,  0.3, 0.7]
])

output = model(X)

print("Output shape:", output.shape)
print(output)

2. Deep Neural Networks (DNN)#

A Deep Neural Network (DNN) is a neural network containing multiple layers of computation, allowing the model to learn a sequence of increasingly complex transformations.

The term deep refers primarily to the number of layers through which information is transformed.

A simple network:

›Input -> Output

has very little depth.

A deeper network:

Architecture & Data Flow
Input
  |
  v
Layer 1
  |
  v
Layer 2
  |
  v
Layer 3
  |
  v
Layer 4
  |
  v
Output

contains substantially more sequential transformations.

2.1 DNN as a Composition of Functions#

A deep network can be viewed mathematically as a composition of functions:

Mathematical Formulation
f(x) = fL(fL-1(...f2(f1(x))...))

Each layer performs one transformation.

For example:

Mathematical Formulation
A1 = f1(X)
A2 = f2(A1)
A3 = f3(A2)
Output = f4(A3)

The complete model is therefore a composition:

Mathematical Formulation
Output = f4(f3(f2(f1(X))))

This composition is one of the fundamental reasons depth is useful.

2.2 MLP vs DNN#

An MLP is a specific type of feedforward architecture, while DNN is a broader description based on depth.

An MLP can be shallow or deep depending on how many hidden layers it contains.

Architecture & Data Flow
MLP
 |
 +-- Shallow MLP
 |
 +-- Deep MLP

A DNN can also use architectures other than a basic fully connected MLP, so the terms should not always be treated as exact synonyms.

2.3 Deep Representation Building#

A deep network can progressively transform its representation:

Architecture & Data Flow
Input Data
    |
    v
Low-Level Representation
    |
    v
Intermediate Representation
    |
    v
Higher-Level Representation
    |
    v
Task-Specific Representation
    |
    v
Output

The exact interpretation of each representation depends on the problem and architecture.

2.4 A Deeper MLP in PyTorch#


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class DeepMLP(nn.Module):
    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(10, 64),
            nn.ReLU(),

            nn.Linear(64, 64),
            nn.ReLU(),

            nn.Linear(64, 32),
            nn.ReLU(),

            nn.Linear(32, 16),
            nn.ReLU(),

            nn.Linear(16, 2)
        )

    def forward(self, x):
        return self.network(x)

model = DeepMLP()

X = torch.randn(8, 10)
output = model(X)

print("Output shape:", output.shape)

3. Network Depth#

Network depth refers to the number of sequential computational layers in a neural network.

Depth measures how many transformations an input passes through before reaching the output.

Architecture & Data Flow
Input
  |
  v
Transformation 1
  |
  v
Transformation 2
  |
  v
Transformation 3
  |
  v
Output

The network becomes deeper as the number of sequential transformations increases.

3.1 Depth in a Dense Network#

Consider:

›Input -> Dense -> ReLU -> Dense -> ReLU -> Dense -> Output

The exact numerical depth depends on the counting convention.

A common convention is to count learnable layers:

text
Dense 1
Dense 2
Dense 3

giving a depth of 3 learnable layers.

Some sources instead count all computational layers or include the input/output layer differently.

Important: always state the counting convention when reporting the depth of a network.

3.2 Depth and Hierarchical Composition#

Increasing depth gives the network more sequential transformations:

Architecture & Data Flow
Shallow:

Input -> Transformation -> Output

versus:

Architecture & Data Flow
Deep:

Input
  |
  v
Transformation 1
  |
  v
Transformation 2
  |
  v
Transformation 3
  |
  v
Transformation 4
  |
  v
Output

The deeper network can compose transformations across more stages.

3.3 Depth Does Not Simply Mean "Better"#

Increasing depth can increase the expressive power of a network, but a deeper network is not automatically better.

A deeper model can introduce:

more computation
more parameters depending on the architecture
greater optimization difficulty
greater memory requirements

This is one reason architectural techniques such as residual connections are useful in deep networks.

3.4 Example: Changing Depth#


🐍 PythonInteractive WebAssembly
import torch.nn as nn

shallow = nn.Sequential(
    nn.Linear(20, 64),
    nn.ReLU(),
    nn.Linear(64, 10)
)

deep = nn.Sequential(
    nn.Linear(20, 64),
    nn.ReLU(),
    nn.Linear(64, 64),
    nn.ReLU(),
    nn.Linear(64, 64),
    nn.ReLU(),
    nn.Linear(64, 10)
)

The second model has more sequential learnable transformations and is therefore deeper under the learnable-layer counting convention.

4. Network Width#

Network width refers to the number of neurons or units in a layer.

For a dense layer:

›Input Features -> 128 Neurons

the width of that layer is 128.

Consider:

Architecture & Data Flow
Input
  |
  v
[64 neurons]
  |
  v
[128 neurons]
  |
  v
[32 neurons]
  |
  v
Output

The hidden-layer widths are:

›64 -> 128 -> 32

4.1 Width vs Depth#

These are different architectural dimensions.

Depth:

›How many layers?

Width:

›How many units are in each layer?

For example:

Architecture & Data Flow
Network A:

Input
  |
  v
[128]
  |
  v
[128]
  |
  v
Output

Depth: smaller
Width: larger

Another network:

Architecture & Data Flow
Network B:

Input
  |
  v
[32]
  |
  v
[32]
  |
  v
[32]
  |
  v
[32]
  |
  v
Output

Depth: larger
Width: smaller

4.2 Effect of Increasing Width#

Increasing width gives a layer more units with which to transform and represent information.

Architecture & Data Flow
Narrow:

Input -> [4 neurons] -> Output

Architecture & Data Flow
Wide:

Input -> [128 neurons] -> Output

A wider layer can represent more features simultaneously, but it also generally increases the number of parameters and computation.

4.4 Depth vs Width#

Property	Depth	Width
Meaning	Number of sequential layers	Number of units in a layer
Controls	Number of transformations	Number of units per transformation
Increasing it	Adds more stages	Adds more units
Typical effect	More hierarchical composition	More representation within a stage
Parameter effect	Usually increases parameters	Usually increases parameters

Depth and width can also be changed independently.

5. Parameter Count#

A parameter is a learnable numerical value adjusted during training.

For a fully connected layer, the main learnable parameters are:

Weights
Biases

5.1 Parameters in a Dense Layer#

Suppose a dense layer has:

Mathematical Formulation
Input features  = n_in
Output neurons  = n_out

Every input feature connects to every output neuron.

Therefore, the number of weights is:

Mathematical Formulation
Number of weights = n_in × n_out

Each output neuron also has one bias:

Mathematical Formulation
Number of biases = n_out

Therefore:

Mathematical Formulation
Total parameters = (n_in × n_out) + n_out

or equivalently:

Mathematical Formulation
Total parameters = n_out × (n_in + 1)

when bias is enabled.

5.2 Example#

Suppose:

Mathematical Formulation
Input features = 4
Output neurons = 5

Weights:

Mathematical Formulation
4 × 5 = 20

Biases:

›5

Total:

Mathematical Formulation
20 + 5 = 25 parameters

5.3 Parameter Count of an Entire MLP#

Consider:

›4 -> 5 -> 3 -> 2

Layer 1:

Mathematical Formulation
4 × 5 + 5 = 25

Layer 2:

Mathematical Formulation
5 × 3 + 3 = 18

Layer 3:

Mathematical Formulation
3 × 2 + 2 = 8

Total:

Mathematical Formulation
25 + 18 + 8 = 51 parameters

5.5 Parameter Count Without Bias#

If a dense layer does not use a bias:

Mathematical Formulation
Parameters = n_in × n_out

For example:

Mathematical Formulation
Input = 4
Output = 5

With bias:
4 × 5 + 5 = 25

Without bias:
4 × 5 = 20

5.6 Parameter Count Using PyTorch#

PyTorch can calculate the number of parameters directly.


🐍 PythonInteractive WebAssembly
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(4, 5),
    nn.ReLU(),
    nn.Linear(5, 3),
    nn.ReLU(),
    nn.Linear(3, 2)
)

total_parameters = sum(
    parameter.numel()
    for parameter in model.parameters()
)

print("Total parameters:", total_parameters)

Output:

›Total parameters: 51

5.7 Parameter Count and Architecture#

For dense networks, increasing width can rapidly increase parameters because adjacent layers are fully connected.

For example:

›Input -> 64 -> 64 -> Output

versus:

›Input -> 256 -> 256 -> Output

The second architecture has many more connections between layers.

Therefore, parameter count is strongly affected by both:

Network depth
Network width

But parameter count alone does not completely describe a model's architecture or expressive ability.

6. Capacity#

Model capacity refers to the ability of a model to represent and learn a wide or complex set of functions.

A model with greater capacity has more flexibility in the functions it can represent.

Capacity is influenced by several factors, including:

Number of parameters
Network depth
Network width
Architecture
Constraints on the model

6.1 Low-Capacity vs High-Capacity Model#

A simple model may have limited capacity:

›Input -> [2 neurons] -> Output

A larger model may have substantially more capacity:

Architecture & Data Flow
Input
  |
  v
[128]
  |
  v
[128]
  |
  v
[128]
  |
  v
Output

The larger network has more learnable parameters and more opportunities to represent complex functions.

6.2 Capacity and Underfitting#

If a model has insufficient capacity for the underlying problem, it may fail to capture important patterns.

This is associated with underfitting.

Conceptually:

Architecture & Data Flow
Too little capacity
       |
       v
Cannot represent the required function well
       |
       v
Underfitting

6.3 Capacity and Overfitting#

A model with very high capacity can potentially fit highly complex patterns, including patterns specific to the training data.

This can contribute to overfitting, especially when the available data, regularization, architecture, or training setup does not adequately constrain the model.

Architecture & Data Flow
Very high capacity
       |
       v
Can represent extremely complex functions
       |
       v
May fit noise or training-specific patterns
       |
       v
Possible overfitting

High capacity does not automatically mean overfitting. Modern neural networks can be highly overparameterized and still generalize well depending on the data, architecture, optimization, and regularization.

6.4 Capacity vs Parameter Count#

Parameter count and capacity are related but are not identical concepts.

Architecture & Data Flow
Parameter Count
      |
      v
One important factor influencing capacity

Two models can have similar numbers of parameters but different architectures and therefore different representational properties.

For example:

text
Model A:
More depth + less width

Model B:
Less depth + more width

They can have comparable parameter counts while using those parameters in very different ways.

6.5 Capacity, Depth, and Width#

A useful conceptual relationship is:

Architecture & Data Flow
Depth  ----\
            \
             ---> Model Capacity
            /
Width  -----/

However, there is no simple universal equation saying that a certain increase in depth or width produces a fixed increase in capacity.

The architecture matters.

7. Residual Connections#

A residual connection is a specific type of skip connection in which the input to a block is added to the block's transformed output.

The basic structure is:

Architecture & Data Flow
             ┌──────────────────────┐
             │                      │
x ---------->│      F(x)            │
|            │                      v
|            └────────────────────> (+) ---> y
|                                     ^
└─────────────────────────────────────┘

Mathematically:

Mathematical Formulation
y = F(x) + x

Here:

x = input to the block
F(x) = transformation performed by the block
y = output after addition

This is called residual learning because the block learns a transformation F(x) that is added to the original input.

7.1 Without a Residual Connection#

A normal stacked block can look like:

Architecture & Data Flow
x
 |
 v
Layer 1
 |
 v
Activation
 |
 v
Layer 2
 |
 v
y

The signal must pass through every transformation.

7.2 With a Residual Connection#

With a residual connection:

Architecture & Data Flow
             ┌───────────────────┐
             │                   │
x ----------> Layer 1 -> Act -> Layer 2
|                                    |
|                                    v
└────────────────────────────────--> (+) -> y

The original input travels through a shortcut path and is added to the transformed path.

7.3 Residual Function#

Instead of directly learning:

Mathematical Formulation
y = H(x)

a residual block learns:

Mathematical Formulation
y = F(x) + x

where:

Mathematical Formulation
F(x) = H(x) - x

The block therefore learns the difference, or residual, relative to the identity mapping.

7.4 Why Residual Connections Help#

As networks become deeper, optimization can become difficult because information and gradients must pass through many transformations.

The shortcut creates a more direct path through the network.

Conceptually:

Architecture & Data Flow
Without shortcut:

x -> Layer -> Layer -> Layer -> Layer -> ...

With shortcut:

x -------------------------------> +
 \-> Layer -> Layer -> Layer ----->

The direct path can make gradient flow and optimization easier.

Residual connections therefore help mitigate optimization difficulties in deep networks.

They do not guarantee that vanishing gradients or other optimization problems disappear completely.

7.5 Gradient Flow#

For:

Mathematical Formulation
y = F(x) + x

the derivative with respect to x is:

Mathematical Formulation
dy/dx = dF(x)/dx + I

where I is the identity transformation.

The important idea is that the gradient contains a direct identity contribution in addition to the gradient through F.

This provides an additional path for gradient propagation.

7.6 Matching Dimensions#

For the simple equation:

Mathematical Formulation
y = F(x) + x

F(x) and x must have compatible shapes.

If the dimensions do not match, a projection can transform x:

Mathematical Formulation
y = F(x) + W_s x

where W_s is a learnable projection.

Conceptually:

Architecture & Data Flow
x -----------------> Projection --------\
                                          (+) -> y
x -> Layer -> Layer --------------------/

7.7 Residual MLP Block Using PyTorch#


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, features):
        super().__init__()

        self.block = nn.Sequential(
            nn.Linear(features, features),
            nn.ReLU(),
            nn.Linear(features, features)
        )

    def forward(self, x):
        return x + self.block(x)

block = ResidualBlock(16)

X = torch.randn(8, 16)
output = block(X)

print("Input shape :", X.shape)
print("Output shape:", output.shape)

The important operation is:


🐍 PythonInteractive WebAssembly
return x + self.block(x)

The original input is preserved through the shortcut path and added to the transformed output.

7.8 Residual Block with Different Dimensions#

If the input and output dimensions differ, a projection can be used:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()

        self.block = nn.Sequential(
            nn.Linear(in_features, out_features),
            nn.ReLU(),
            nn.Linear(out_features, out_features)
        )

        self.shortcut = (
            nn.Identity()
            if in_features == out_features
            else nn.Linear(in_features, out_features)
        )

    def forward(self, x):
        return self.shortcut(x) + self.block(x)

block = ResidualBlock(16, 32)

X = torch.randn(8, 16)
output = block(X)

print(output.shape)

Here:

Architecture & Data Flow
Input:      16 features
Main path:  16 -> 32 -> 32
Shortcut:   16 -> 32
Output:     32 features

Both paths therefore produce compatible tensors before addition.

8. Skip Connections#

A skip connection is a connection that bypasses one or more intermediate layers and sends information directly to a later layer.

General structure:

Architecture & Data Flow
x
| \
|  \
|   v
|  Layer 1
|    |
|    v
|  Layer 2
|    |
|    v
|   (+ / concat)
|      ^
└──────┘

The defining idea is the shortcut path.

8.1 Purpose of Skip Connections#

Skip connections provide an alternative path for information to travel through the network.

Instead of forcing information to pass through every intermediate transformation:

›x -> Layer 1 -> Layer 2 -> Layer 3 -> y

a shortcut can provide:

Mathematical Formulation
x -------------------------------> y
 \-> Layer 1 -> Layer 2 -> Layer 3 /

This can improve information flow and make optimization of deeper networks easier.

8.2 Skip Connections Are a Broader Concept#

Residual connections are a type of skip connection.

The relationship is:

Architecture & Data Flow
Skip Connections
       |
       +---- Residual Connections

A skip connection describes the general idea of bypassing layers.

A residual connection specifically uses an additive operation:

Mathematical Formulation
y = F(x) + x

8.3 Additive Skip Connection#

An additive skip connection combines the shortcut and transformed paths by addition:

Architecture & Data Flow
Main path:     F(x)
                  \
                   (+) -> y
                  /
Shortcut:       x

Formula:

Mathematical Formulation
y = F(x) + x

This is the standard residual form.

8.4 Concatenative Skip Connection#

A skip connection can also combine representations by concatenation rather than addition.

Conceptually:

Architecture & Data Flow
Main path:     F(x)
                  \
                   [ Concatenate ] -> y
                  /
Shortcut:       x

Instead of:

Mathematical Formulation
y = F(x) + x

the representations are joined along a feature/channel dimension.

The resulting representation therefore contains information from both paths.

8.5 Skip Connection vs Residual Connection#

Property	Skip Connection	Residual Connection
Meaning	General shortcut connection	Specific type of skip connection
Bypasses layers	Yes	Yes
Combination	Can use addition, concatenation, or another mechanism	Typically addition
Basic form	`x -> later layer`	`y = F(x) + x`
Main purpose	Improve information/gradient flow	Improve information/gradient flow and deep-network optimization

Therefore:

text
Every residual connection is a skip connection,
but not every skip connection is a residual connection.

8.6 Skip Connections in a Deep Feedforward Network#

A conventional deep network:

Architecture & Data Flow
x
 |
 v
Layer 1
 |
 v
Layer 2
 |
 v
Layer 3
 |
 v
Layer 4
 |
 v
y

A network with a shortcut:

Architecture & Data Flow
x
| \
|  \
|   v
| Layer 1
|   |
|   v
| Layer 2
|   |
|   v
| Layer 3
|   |
|   v
| Layer 4
|   |
|   v
|   y
|  /
└─/

The shortcut allows the original information to bypass intermediate transformations.

8.7 Simple Skip Connection in PyTorch#


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn

class SkipBlock(nn.Module):
    def __init__(self, features):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(features, features),
            nn.ReLU(),
            nn.Linear(features, features)
        )

    def forward(self, x):
        transformed = self.layers(x)

        # Skip connection
        output = transformed + x

        return output

block = SkipBlock(32)

X = torch.randn(4, 32)
output = block(X)

print("Output shape:", output.shape)

Because the example combines the tensors using addition, this particular skip connection is also a residual connection.

Summary Table#

Concept	Core Idea
MLP	Feedforward network primarily composed of fully connected layers
DNN	Neural network with multiple layers of computation
Depth	Number of sequential transformations/layers
Width	Number of neurons/units in a layer
Parameter Count	Total number of learnable weights and biases
Capacity	Ability of a model to represent complex functions
Skip Connection	Shortcut that bypasses one or more layers
Residual Connection	Additive skip connection, typically `y = F(x) + x`

Quick Recap#

An MLP is a feedforward architecture built primarily from fully connected layers.
A DNN uses multiple layers to compose a sequence of transformations.
Depth measures the number of sequential layers or transformations, depending on the counting convention.
Width measures the number of neurons or units in a layer.
A dense layer with n_in inputs and n_out outputs has n_in × n_out + n_out parameters when bias is enabled.
Capacity describes how complex a function a model can represent and is influenced by architecture, depth, width, and parameterization.
A skip connection provides a shortcut around one or more layers.
A residual connection is an additive skip connection represented by y = F(x) + x.
Residual connections provide a more direct information and gradient path, helping optimization in deep networks.
Parameter count and capacity are related, but they are not the same thing.

Knowledge Checkpoint

10. Deep Feedforward Networks Checkpoint

Q1.What does the Universal Approximation Theorem state about Multi-Layer Perceptrons?

AA feedforward network with a single hidden layer and non-linear activation can approximate any continuous function on compact subsets of R^n to arbitrary precision, given sufficient width.

BAny neural network will reach 100% test accuracy on any dataset.

CDeep networks always train faster than shallow networks.

DLinear activation functions can model non-linear boundaries if depth >= 10.

Q2.Why are deep networks (greater depth) practically superior to extremely wide shallow networks?

ADepth allows hierarchical feature composition (reusing lower-level features in higher abstractions), requiring exponentially fewer parameters than shallow networks.

BShallow networks cannot be trained with GPUs.

CDeep networks have zero risk of overfitting.

DDeep networks don't require activation functions.

Q3.How do residual connections (y = F(x) + x) resolve the degradation problem in very deep feedforward networks?

AThey create an identity shortcut path that allows gradients to flow directly back through layers without attenuation (dL/dx = dL/dy * (dF/dx + 1)).

BThey double the number of parameters per layer.

CThey force all activations to be zero.

DThey prevent GPU memory overflow.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

09. Normalization (BatchNorm, LayerNorm, RMSNorm, GroupNorm)

Next Lesson

11. Convolutional Neural Networks (CNN) Fundamentals