Advanced

120–180 min read

#Transformers#Attention#Self-Attention#Query Key Value#Multi-Head Attention#Positional Encoding#Token Embeddings#Causal Language Modeling#GPT#BERT#T5#Llama#LLM Architecture

Transformers & How Large Language Models Work

A beginner-friendly to advanced guide to Transformer architecture and the internal mechanics of modern Large Language Models, including attention, token embeddings, causal language modeling, Transformer blocks, and major model architectures.

Transformers & How Large Language Models Work

1. Introduction#

In the previous section, we learned:

What Generative AI is
What an LLM is
Tokens
Context windows
Training
Fine-tuning
Inference
Hallucinations
RAG
Tool calling
Agents
LangChain
LangGraph

Now we go one level deeper.

The central question of this notebook is:

What actually happens inside an LLM when we give it a prompt?

Modern LLMs are largely built around the Transformer architecture.

Understanding Transformers is extremely useful because it explains why modern systems can:

Process long sequences
Understand relationships between words
Generate coherent text
Handle different languages
Perform summarization
Generate code
Support reasoning-like behavior
Power RAG systems
Support tool calling and agents

This notebook progresses from intuition to mathematical foundations and finally to practical implementation.

2. Learning Objectives

By the end of this notebook, you should be able to:

Explain why sequence modeling is difficult.
Explain the limitations of traditional RNN-based approaches.
Explain the motivation behind attention.
Understand self-attention.
Understand Query, Key, and Value.
Calculate attention conceptually.
Understand scaled dot-product attention.
Understand multi-head attention.
Understand positional information.
Explain token embeddings.
Understand Transformer blocks.
Understand residual connections.
Understand layer normalization.
Understand feed-forward networks.
Explain encoder and decoder architectures.
Understand causal language modeling.
Explain why GPT-style models use causal masking.
Compare BERT, GPT, and T5-style architectures.
Understand decoder-only architectures such as Llama-style models.
Trace a prompt through an LLM.
Build a small Transformer component using PyTorch.
Understand the difference between training and generation.
Understand the major computational challenges of Transformers.

3. Why Do We Need Sequence Models?

Language is sequential.

Consider:

›The cat sat on the mat.

The meaning of a word depends heavily on its surrounding context.

For example:

›bank

could refer to:

›river bank

or:

›bank account

The surrounding tokens help determine the meaning.

Therefore, language models need mechanisms for understanding relationships between tokens.

4. Early Approach: Recurrent Neural Networks

Before Transformers became dominant, recurrent neural networks were widely used for sequence modeling.

A simplified RNN looks like:

text
x1 → RNN → h1
 ↓
x2 → RNN → h2
 ↓
x3 → RNN → h3
 ↓
x4 → RNN → h4

The hidden state carries information from previous time steps.

A simplified equation is:

h_t = f(W_xx_t + W_hh_{t-1} + b)

where:

$x_t$ = current input
$h_t$ = current hidden state
$h_{t-1}$ = previous hidden state
$W_x$ = input weights
$W_h$ = recurrent weights
$b$ = bias

5. Problems With RNNs

RNNs have important limitations.

Sequential computation#

The next step depends on the previous hidden state.

This makes large-scale parallel processing difficult.

Long-range dependencies#

Information from very early tokens can become difficult to preserve across a long sequence.

Vanishing gradients#

During backpropagation through many time steps, gradients can become extremely small.

Exploding gradients#

Gradients can also become extremely large.

LSTMs and GRUs were developed to address some of these problems.

But another major idea eventually changed sequence modeling:

Attention.

6. The Attention Idea

Suppose we are processing:

›The animal didn't cross the road because it was tired.

What does:

›it

refer to?

The model needs to determine which earlier tokens are relevant.

Attention provides a mechanism for a token to consider other tokens.

Conceptually:

text
Current token
 ↓
Look at other tokens
 ↓
Assign importance
 ↓
Combine useful information

Instead of forcing all information through one recurrent hidden state, attention creates direct relationships between positions.

7. Self-Attention

Self-attention means that tokens in the same sequence attend to one another.

For:

›The cat sat on the mat

the representation of:

›cat

can attend to:

text
The
sat
mat

and other relevant positions.

Conceptually:

text
The ─────┐
cat ─────┤
sat ─────┼──→ Self-Attention
on ──────┤
the ─────┤
mat ─────┘

Each token can build a context-aware representation.

8. Query, Key, and Value

Self-attention uses three representations:

Query (Q)
Key (K)
Value (V)

A useful analogy is a search system.

Query#

What information am I looking for?

Key#

What information does each token represent for matching?

Value#

What information should be retrieved if the token is relevant?

For every input representation, the model learns transformations that produce:

text
Q
K
V

9. Creating Q, K, and V

Suppose the input representation is:

X

The model applies learned matrices:

Q = XW_Q

K = XW_K

V = XW_V

where:

$W_Q$ = learned query projection
$W_K$ = learned key projection
$W_V$ = learned value projection

These matrices are learned during training.

10. Attention Scores

The model compares a query with keys using a dot product.

Conceptually:

score(Q,K) = QK^T

A larger score means stronger similarity between a query and a key.

For example:

text
Query: "it"

Key: "animal" → high score
Key: "road" → lower score
Key: "because" → lower score

The actual values are learned and depend on the model's internal representations.

11. Scaling

The dot product is divided by the square root of the key dimension:

\frac{QK^T}{\sqrt{d_k}}

This prevents values from becoming excessively large as dimensionality increases.

12. Softmax

The scaled scores are passed through softmax:

AttentionWeights = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)

Softmax converts the scores into a probability-like distribution.

For example:

text
Token A → 0.10
Token B → 0.65
Token C → 0.25

The weights indicate how strongly information from each position contributes.

13. Weighted Values

The attention weights are multiplied by the values:

Attention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V

The resulting representation combines information from relevant tokens.

This is the core equation of scaled dot-product attention.

14. Attention Example

Suppose:

›The cat sat on the mat

When processing:

›cat

the model might learn an attention pattern such as:

text
The → 0.05
cat → 0.10
sat → 0.40
on → 0.05
the → 0.10
mat → 0.30

These numbers are only illustrative.

The important concept is:

Different tokens can receive different levels of attention.

15. Self-Attention Pipeline

The complete conceptual process is:

text
Input representations
 ↓
Create Q, K, V
 ↓
Q × Kᵀ
 ↓
Scale by √dₖ
 ↓
Apply mask if required
 ↓
Softmax
 ↓
Weighted sum of V
 ↓
Attention output

This operation is repeated across the sequence.

16. Why Attention Is Powerful

Attention allows the model to create direct relationships between tokens.

Compare:

text
RNN:

Token 1 → Token 2 → Token 3 → Token 4

with:

text
Self-Attention:

Token 1 ─┐
Token 2 ─┼─→ Every relevant token can interact
Token 3 ─┤
Token 4 ─┘

This makes long-range relationships easier to model.

17. Multi-Head Attention

One attention operation may not be enough.

Different attention heads can learn different relationships.

For example:

text
Head 1 → grammatical relationships
Head 2 → nearby context
Head 3 → long-range relationships
Head 4 → semantic relationships

These are conceptual examples rather than guaranteed interpretations of individual heads.

18. Multi-Head Attention Process

Conceptually:

text
Input
 ↓
 ┌────────┬────────┬────────┬────────┐
 ↓ ↓ ↓ ↓
Head 1 Head 2 Head 3 Head 4
 ↓ ↓ ↓ ↓
 └────────┴────────┴────────┴────────┘
 ↓
 Concatenate
 ↓
 Linear projection
 ↓
 Output

Each head has its own learned projections.

19. Multi-Head Attention Equations

For head $i$ :

head_i = Attention( XW_Q^i, XW_K^i, XW_V^i )

Then:

MultiHead(X) = Concat(head_1,\ldots,head_h)W_O

where:

$h$ = number of heads
$W_O$ = output projection

20. Positional Information

Attention alone does not inherently encode the order of tokens.

Consider:

›Dog bites man.

versus:

›Man bites dog.

The same words can produce a completely different meaning when their order changes.

Therefore, Transformers need positional information.

21. Positional Encoding

The original Transformer introduced positional encodings.

A commonly described sinusoidal formulation is:

PE(pos,2i) = \sin \left( \frac{pos}{10000^{2i/d}} \right)

and:

PE(pos,2i+1) = \cos \left( \frac{pos}{10000^{2i/d}} \right)

The positional representation is combined with token representations.

Conceptually:

text
Token embedding
 +
Position information
 ↓
Transformer input

Modern architectures may use other positional mechanisms, including learned or relative-position approaches and rotary positional embeddings.

22. Token Embeddings

Before entering a Transformer, token IDs are converted into vectors.

For example:

text
Token ID
 ↓
Embedding lookup
 ↓
Vector

Conceptually:

text
"cat"
 ↓
[0.21, -0.14, 0.73, ...]

The actual embedding dimensions can be hundreds or thousands.

23. Embedding Matrix

Suppose:

Mathematical Formulation
Vocabulary size = V
Embedding dimension = D

The model can maintain an embedding matrix:

E \in \mathbb{R}^{V \times D}

Each token ID selects one row.

text
Token ID
 ↓
Embedding matrix
 ↓
Token vector

These embeddings are learned during training.

24. From Tokens to Transformer Input

The high-level pipeline is:

text
Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Token Embeddings
 ↓
Positional Information
 ↓
Transformer

This is the first major transformation of a prompt inside an LLM.

25. The Transformer Block

A typical Transformer block contains:

text
Input
 ↓
Self-Attention
 ↓
Residual Connection
 ↓
Layer Normalization
 ↓
Feed-Forward Network
 ↓
Residual Connection
 ↓
Layer Normalization
 ↓
Output

Exact ordering varies between architectures.

26. Residual Connections

Residual connections help information flow through deep neural networks.

A simplified equation is:

Output = x + F(x)

Instead of replacing the original representation completely, the network learns an update.

Conceptually:

text
Input ───────────────┐
 ↓ │
Attention → Update ──┤
 ↓
 Add

Residual connections are important for training deep Transformer networks.

27. Layer Normalization

Layer normalization stabilizes neural network computation.

A simplified form is:

LayerNorm(x) = \gamma \frac{x-\mu} {\sqrt{\sigma^2+\epsilon}} +\beta

where:

$\mu$ = mean
$\sigma^2$ = variance
$\epsilon$ = small numerical constant
$\gamma,\beta$ = learned parameters

The exact architecture can use pre-normalization or post-normalization.

28. Feed-Forward Network

After attention, each token representation passes through a feed-forward network.

A simplified form is:

FFN(x) = W_2 \sigma(W_1x+b_1)+b_2

where $\sigma$ is an activation function.

Modern Transformer architectures can use activations such as:

ReLU
GELU
SwiGLU and related gated functions

The feed-forward layer adds nonlinear transformation capacity.

29. Transformer Block Intuition

Think of a Transformer block as two major operations:

text
1. Attention
 "Which other tokens are relevant?"

2. Feed-forward network
 "How should I transform the resulting representation?"

Repeated blocks gradually transform token representations into increasingly useful internal representations.

30. Stacking Transformer Blocks

A large model contains many Transformer blocks.

Conceptually:

text
Input
 ↓
Block 1
 ↓
Block 2
 ↓
Block 3
 ↓
...
 ↓
Block N
 ↓
Output representation

The exact number of blocks depends on the model architecture.

31. Encoder Architecture

The original Transformer architecture contains an encoder and decoder.

The encoder processes an input sequence and creates contextual representations.

Conceptually:

text
Input
 ↓
Embedding
 ↓
Encoder Block
 ↓
Encoder Block
 ↓
Encoder Output

Encoder models are useful for understanding or representing input sequences.

32. Decoder Architecture

The decoder generates output tokens.

Conceptually:

text
Previous output tokens
 ↓
 Decoder
 ↓
Next token

Decoder architectures use causal masking during autoregressive generation.

33. Encoder-Decoder Architecture

The original Transformer uses:

text
Input
 ↓
Encoder
 ↓
Context representations
 ↓
Decoder
 ↓
Output

This is useful for sequence-to-sequence tasks such as:

text
English → French
Document → Summary
Question → Answer

34. Causal Language Modeling

GPT-style language models commonly use causal language modeling.

The objective is:

Predict the next token using only previous tokens.

For:

›The cat sat

the model predicts:

›on

Then:

›The cat sat on

predicts the next token.

35. Causal Masking

During training, a token should not see future tokens.

For example:

text
Token 1 → can see Token 1
Token 2 → can see Token 1–2
Token 3 → can see Token 1–3
Token 4 → can see Token 1–4

Conceptually:

text
 1 2 3 4
1 X X X
2 X X
3 X
4

This is the causal attention mask.

It prevents information leakage from the future during training.

36. Why Causal Masking Matters

Without masking, the model could simply look at the answer token while training.

That would make the training task unrealistic.

Causal masking ensures:

›Past → Current prediction

rather than:

›Past + Future → Current prediction

This is fundamental to autoregressive language modeling.

37. GPT-Style Models

GPT-style architectures are generally decoder-only Transformers.

Conceptually:

text
Text
 ↓
Tokenizer
 ↓
Token embeddings
 ↓
Positional mechanism
 ↓
Decoder-only Transformer blocks
 ↓
Language-model head
 ↓
Next-token probabilities

They are especially suitable for autoregressive generation.

38. Language Model Head

After the final Transformer block, the model needs to convert hidden representations into vocabulary scores.

Conceptually:

text
Final hidden state
 ↓
Linear projection
 ↓
Vocabulary logits
 ↓
Softmax
 ↓
Token probabilities

If the vocabulary contains:

›50,000 tokens

the output can contain one score for each token.

39. Weight Tying

Some language models tie the token embedding matrix and output projection weights.

Conceptually:

text
Input embedding
 ↕
Shared weights
 ↕
Output projection

This can reduce the number of independent parameters and sometimes improve parameter efficiency.

Not every architecture uses weight tying.

40. BERT-Style Models

BERT is primarily an encoder-style Transformer architecture.

Its original training approach includes masked language modeling.

For example:

›The cat [MASK] on the mat.

The model learns to predict the masked token using surrounding context.

This differs from causal language modeling:

text
GPT:
Use previous tokens to predict next token.

BERT:
Use surrounding context to predict masked tokens.

BERT-style models are commonly associated with understanding tasks rather than unrestricted autoregressive text generation.

41. T5-Style Models

T5 uses an encoder-decoder Transformer architecture and frames many NLP tasks as text-to-text problems.

Conceptually:

text
Input text
 ↓
Encoder
 ↓
Decoder
 ↓
Output text

Examples:

text
Translate English to French
Summarize document
Answer question

This architecture is useful when input and output are both sequences.

42. Llama-Style Decoder Architectures

Modern open-weight decoder models commonly use decoder-only Transformer designs.

A simplified architecture is:

text
Tokens
 ↓
Embeddings
 ↓
Positional mechanism
 ↓
Repeated decoder blocks
 ↓
Normalization
 ↓
LM head
 ↓
Next-token probabilities

Specific implementations can include architectural improvements such as:

Rotary positional embeddings
RMSNorm
Gated feed-forward networks
Grouped-query attention

The exact architecture depends on the model version.

43. Rotary Positional Embeddings

Rotary positional embeddings, often abbreviated as RoPE, encode position information by rotating components of query and key representations.

Conceptually:

text
Query + position
Key + position
 ↓
Position-aware attention

RoPE is widely used in modern decoder architectures.

Its practical goal is to incorporate relative positional information into attention calculations.

44. RMSNorm

Some modern architectures use RMSNorm rather than traditional LayerNorm.

A simplified form is:

RMSNorm(x) = \frac{x} {\sqrt{ \frac{1}{d}\sum_i x_i^2+\epsilon }} \odot g

RMSNorm omits the mean-centering step used by LayerNorm.

Again, exact implementation depends on the architecture.

45. Grouped-Query Attention

Attention can become expensive for large models.

Grouped-query attention (GQA) reduces some key/value memory requirements by allowing multiple query heads to share key/value heads.

Conceptually:

text
Many Query heads
 ↓
Shared groups of
Key/Value heads

This can improve inference efficiency while retaining many benefits of multi-head attention.

46. The Full LLM Pipeline

We can now combine the concepts.

text
User prompt
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Token embeddings
 ↓
Positional information
 ↓
Transformer block
 ↓
Transformer block
 ↓
...
 ↓
Final hidden states
 ↓
Language-model head
 ↓
Logits
 ↓
Sampling / decoding
 ↓
Next token
 ↓
Repeat
 ↓
Generated response

This is one of the most important diagrams in the entire Generative AI course.

47. What Happens During One Generation Step?

Suppose the prompt is:

›The capital of France is

The model processes the token sequence.

The final representation is converted into logits.

Conceptually:

text
Paris → high probability
London → low probability
Berlin → low probability
...

A decoding method selects the next token:

›Paris

Now the sequence becomes:

›The capital of France is Paris

The model runs again to predict the next token.

48. Why Generation Is Repeated

Autoregressive generation is sequential.

text
Prompt
 ↓
Predict token 1
 ↓
Predict token 2
 ↓
Predict token 3
 ↓
...

This is why generating long outputs can take substantial time.

Modern inference systems use optimization techniques such as:

KV caching
batching
quantization
optimized kernels
speculative decoding
parallel serving

49. KV Cache

During autoregressive generation, previous key and value representations do not always need to be recomputed from scratch.

A KV cache stores them.

Conceptually:

text
Previous tokens
 ↓
Cached K/V
 ↓
New token
 ↓
Attention

This can significantly improve generation efficiency.

The cache also consumes memory, especially for long contexts and large models.

50. Training vs Inference Inside the Transformer

Training#

Many tokens can be processed in parallel because the complete training sequence is available.

Causal masking prevents future-token leakage.

text
Full sequence
 ↓
Masked attention
 ↓
Next-token predictions

Inference#

Future tokens do not yet exist.

The model generates one or more tokens and uses the generated tokens as new context.

text
Prompt
 ↓
Token
 ↓
Token
 ↓
Token

This distinction is critical for understanding LLM performance.

51. Why Training Can Be More Parallel

During training, suppose we have:

›The cat sat on the mat

The model can calculate many next-token predictions in a single forward pass using masking.

For example:

text
The → cat
The cat → sat
The cat sat → on
The cat sat on → the

The causal mask prevents each position from seeing future information.

This makes training much more parallelizable than autoregressive inference.

52. Computational Complexity of Attention

For standard self-attention, the attention matrix grows approximately as:

O(T^2)

where:

$T$ = sequence length

If sequence length doubles:

›T → 2T

the attention interaction matrix grows approximately:

›T² → 4T²

This is one reason long-context inference can be computationally expensive.

53. Memory Complexity

Large models require memory for:

Model parameters
Activations
KV cache
Gradients during training
Optimizer states during training

Training generally requires much more memory than inference.

This is one reason large-scale model training requires specialized infrastructure.

54. Scaling a Transformer

Model capacity can be increased through combinations of:

More layers
Larger hidden dimensions
More attention heads
Larger training datasets
More training compute

But simply increasing everything is not automatically optimal.

Efficient scaling requires balancing:

text
Parameters
+
Data
+
Compute
+
Architecture

55. Dense vs Mixture-of-Experts Models

Not every modern language model activates all parameters for every token.

A dense model:

text
Token
 ↓
All model layers/parameters

A Mixture-of-Experts (MoE) architecture can route tokens to selected expert networks.

Conceptually:

text
 Router
 ↓
 ┌──────┼──────┐
 ↓ ↓ ↓
 Expert Expert Expert
 └──────┼──────┘
 ↓
 Output

This can increase total parameter capacity while limiting the amount of computation used for each token.

The routing strategy and exact architecture vary by model.

56. Practical PyTorch: Scaled Dot-Product Attention

A simplified implementation:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn.functional as F


def scaled_dot_product_attention(q, k, v, mask=None):
 d_k = q.size(-1)

 scores = torch.matmul(
 q,
 k.transpose(-2, -1)
 ) / (d_k ** 0.5)

 if mask is not None:
 scores = scores.masked_fill(
 mask == 0,
 float("-inf")
 )

 weights = F.softmax(
 scores,
 dim=-1
 )

 output = torch.matmul(
 weights,
 v
 )

 return output, weights

This implementation demonstrates the core mathematical idea.

Production implementations are usually highly optimized.

57. Practical PyTorch: Multi-Head Attention

PyTorch provides an implementation:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn


attention = nn.MultiheadAttention(
 embed_dim=128,
 num_heads=4,
 batch_first=True
)

Example:


🐍 PythonInteractive WebAssembly
x = torch.randn(
 8,
 32,
 128
)

output, weights = attention(
 x,
 x,
 x
)

The shape is:

›(batch, sequence_length, embedding_dimension)

58. Practical PyTorch: Transformer Encoder Layer


🐍 PythonInteractive WebAssembly
encoder_layer = nn.TransformerEncoderLayer(
 d_model=128,
 nhead=4,
 batch_first=True
)

x = torch.randn(
 8,
 32,
 128
)

output = encoder_layer(x)

This gives a practical way to experiment with Transformer components without implementing every operation manually.

59. Building a Tiny Decoder-Style Model

A simplified educational architecture:


🐍 PythonInteractive WebAssembly
import torch
import torch.nn as nn


class TinyLanguageModel(nn.Module):
 def __init__(
 self,
 vocab_size,
 d_model=128,
 nhead=4,
 num_layers=2
 ):
 super().__init__()

 self.embedding = nn.Embedding(
 vocab_size,
 d_model
 )

 layer = nn.TransformerEncoderLayer(
 d_model=d_model,
 nhead=nhead,
 batch_first=True
 )

 self.transformer = nn.TransformerEncoder(
 layer,
 num_layers=num_layers
 )

 self.lm_head = nn.Linear(
 d_model,
 vocab_size
 )

 def forward(self, input_ids):
 x = self.embedding(input_ids)

 x = self.transformer(x)

 logits = self.lm_head(x)

 return logits

For true causal language modeling, the attention mask must prevent each position from seeing future positions.

60. Causal Mask Example

Conceptually:


🐍 PythonInteractive WebAssembly
seq_len = 5

mask = torch.tril(
 torch.ones(
 seq_len,
 seq_len
 )
)

This creates:

This allows each position to attend only to itself and previous positions.

61. Training Objective

For a causal language model, input and target sequences are shifted.

Example:

text
Input:
The cat sat on

Target:
cat sat on the

The model learns:

text
The → cat
The cat → sat
The cat sat → on
The cat sat on → the

A cross-entropy loss can be used:


🐍 PythonInteractive WebAssembly
loss_fn = nn.CrossEntropyLoss()

The model is trained to assign high probability to the correct next token.

62. Why Transformers Changed NLP

Transformers provided several major advantages:

Better parallelization during training
Strong long-range dependency modeling
Scalable architecture
Flexible attention mechanisms
Transfer learning
Large-scale pre-training

This enabled the development of increasingly capable foundation models.

63. Transformer Family Comparison

Architecture	Main structure	Typical strength
Encoder-only	Encoder	Representation / understanding
Decoder-only	Decoder	Autoregressive generation
Encoder-decoder	Both	Sequence-to-sequence transformation
MoE Transformer	Routed experts	Large capacity with selective computation

Examples:

Model family	Architecture style
BERT	Encoder-only
GPT-style	Decoder-only
T5-style	Encoder-decoder
Llama-style	Decoder-only

Architecture names and implementation details can vary across model generations.

64. How This Connects to Generative AI

Now connect the architecture to applications.

text
Transformer
 ↓
LLM
 ↓
Text generation
 ↓
Prompt engineering
 ↓
Structured output
 ↓
RAG
 ↓
Tool calling
 ↓
Agents
 ↓
LangChain
 ↓
LangGraph

The frameworks we will use later sit above these underlying model capabilities.

65. Important Distinction: Model vs Framework

The Transformer is part of the model architecture.

LangChain and LangGraph are application frameworks.

Think of the stack as:

text
Application
 ↓
LangGraph / LangChain
 ↓
RAG / Tools / Memory
 ↓
LLM API or Local Model
 ↓
Transformer Architecture
 ↓
GPU / CPU

This distinction will help you avoid treating a framework abstraction as the model itself.

66. Common Misconceptions

"Attention means the model understands like a human."#

Not necessarily.

Attention is a mathematical mechanism for weighting information between representations.

"Every attention head has a simple human-readable meaning."#

Not necessarily.

Individual heads can learn complex and overlapping patterns.

"Transformers remember everything permanently."#

No.

The model processes the context supplied to it and uses learned parameters.

"A larger context window means perfect memory."#

No.

Longer context increases available input, but does not guarantee perfect retrieval or reasoning.

"More parameters always means better performance."#

No.

Architecture, data, training, evaluation, and task fit all matter.

67. Exercises

Exercise 1: Attention by Hand#

Given:

Mathematical Formulation
Q = [1, 0]
K1 = [1, 0]
K2 = [0, 1]
V1 = [10, 0]
V2 = [0, 20]

Calculate:

Dot products
Scaled scores
Softmax attention weights
Final weighted value

Use this to understand the mechanics of attention.

Exercise 2: Causal Mask#

For a sequence of length 6, create a causal attention mask.

Verify that:

text
Token 1 sees only token 1
Token 2 sees tokens 1–2
...
Token 6 sees tokens 1–6

Exercise 3: Compare Architectures#

Explain the difference between:

text
BERT
GPT
T5
Llama-style decoder model

Focus on:

Architecture
Training objective
Typical use case
Generation capability

Exercise 4: Transformer Block#

Draw the flow of one Transformer block including:

Attention
Residual connection
Normalization
Feed-forward network
Second residual connection
Second normalization

Exercise 5: Generation#

Explain what happens internally when the model receives:

›The weather today is

and generates:

›sunny

Describe:

Tokenization
Embedding
Transformer processing
Logits
Probability distribution
Decoding
New token

68. Mini Project

Build a Tiny Transformer Language Model#

Build a small educational language model using PyTorch.

Requirements:

Create a small text dataset.
Tokenize it.
Build a vocabulary.
Create input/target sequences.
Add token embeddings.
Add positional information.
Add Transformer layers.
Apply causal masking.
Add a language-model head.
Train using cross-entropy loss.
Generate text autoregressively.

Suggested architecture:

text
Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Embedding
 ↓
Positional Information
 ↓
Transformer Blocks
 ↓
Linear LM Head
 ↓
Logits
 ↓
Next Token

Keep the model intentionally small.

The goal is understanding, not competitive language modeling.

69. Advanced Discussion: Why LLMs Can Generalize

A major research question is why large neural language models can perform tasks that were not explicitly trained as separate supervised tasks.

Possible contributing factors include:

Large-scale pre-training
Rich internal representations
Transformer architecture
Diverse training data
Scale
Instruction tuning
In-context learning

The exact mechanisms behind emergent capabilities remain an active research area.

70. In-Context Learning

An LLM can sometimes learn the pattern of a task from examples included in the prompt.

For example:

text
Input: happy
Output: positive

Input: terrible
Output: negative

Input: excellent
Output:

The model may infer:

›positive

without changing its parameters.

This is called in-context learning.

It is different from fine-tuning.

71. Fine-Tuning vs In-Context Learning

In-context learning#

text
Examples
 ↓
Prompt
 ↓
Model
 ↓
Response

No model parameters are changed.

Fine-tuning#

text
Training examples
 ↓
Optimization
 ↓
Updated parameters

The model parameters are changed.

This distinction is fundamental in GenAI engineering.

72. Attention and Long Context

Attention makes it possible for tokens to interact across a sequence.

However, longer context introduces:

More computation
More memory use
Larger KV caches
Potential retrieval problems
Possible degradation in practical usefulness

Therefore:

A large context window is a capability, not a guarantee that every piece of context will be used equally well.

This becomes especially important for RAG systems.

73. Why RAG Still Matters With Long Context

Even if a model supports a large context window, applications may still use retrieval.

Instead of sending:

›10,000 documents

the application can retrieve:

›Top 5 relevant chunks

and provide those to the model.

This can reduce:

Input size
Cost
Latency
Distracting information

Retrieval therefore remains an important architecture even as context windows become larger.

74. What You Should Understand Before Moving On

At this point, you should be able to explain:

text
Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Position
 ↓
Q/K/V
 ↓
Attention
 ↓
Multi-Head Attention
 ↓
Feed-Forward Network
 ↓
Residual + Normalization
 ↓
Repeated Transformer Blocks
 ↓
Logits
 ↓
Next Token

If you understand this pipeline, you have the foundation required to understand modern LLM application frameworks.

75. Final Summary

The Transformer architecture changed modern NLP because it provided a scalable way to model relationships between tokens using attention.

The central concepts are:

text
Tokenization
 ↓
Embeddings
 ↓
Positional Information
 ↓
Self-Attention
 ↓
Multi-Head Attention
 ↓
Feed-Forward Network
 ↓
Residual Connections
 ↓
Normalization
 ↓
Repeated Transformer Blocks
 ↓
Language Model Head
 ↓
Logits
 ↓
Decoding

Different Transformer architectures specialize in different tasks:

text
BERT
→ Encoder-oriented understanding

GPT
→ Decoder-only autoregressive generation

T5
→ Encoder-decoder sequence transformation

Llama-style models
→ Decoder-only generative modeling

And the modern GenAI application stack builds on top:

text
Transformer / LLM
 ↓
Prompting
 ↓
Structured Outputs
 ↓
Embeddings
 ↓
RAG
 ↓
Tools
 ↓
Agents
 ↓
LangChain
 ↓
LangGraph
 ↓
Production GenAI

The key lesson is:

LangChain and LangGraph operate at the application layer. Understanding Transformers gives you the foundation to understand what the underlying LLM is actually doing.

76. Next Notebook

The next notebook is:

Notebook 3 — Prompt Engineering & Structured Outputs

It will progress from beginner techniques to advanced LLM interaction patterns:

What makes a good prompt
System, user, and assistant messages
Instruction hierarchy
Zero-shot prompting
Few-shot prompting
Role and task prompting
Context and constraints
Prompt templates
Delimiters
Output formatting
JSON generation
Structured outputs
Pydantic schemas
Function/tool calling
Prompt chaining
Task decomposition
Query rewriting
Prompt injection
Prompt security
Prompt evaluation
Reusable prompt patterns
Practical Python examples
LangChain prompt templates
Structured-output mini projects

The next notebook will begin the transition from understanding LLM internals to building reliable LLM applications.

Knowledge Checkpoint

LLM Transformer Architectures Checkpoint

Q1.Why are modern state-of-the-art LLMs (e.g. LLaMA, GPT-4, Mistral) primarily Decoder-Only architectures rather than Encoder-Decoder?

ADecoder-only models with causal attention are computationally more efficient for autoregressive pretraining at scale and naturally handle both generation and in-context learning.

BDecoder-only models cannot process text prompts.

CEncoder-decoder models cannot run on GPUs.

DDecoder-only models require zero training data.

Q2.How does FlashAttention optimize the standard self-attention computation?

ABy tiling the attention computation across GPU SRAM and fusing softmax and matrix multiplication, minimizing high-latency reads/writes to High Bandwidth Memory (HBM).

BBy reducing attention precision to 1-bit integers.

CBy discarding 50% of the attention weights.

DBy computing attention on CPU.

Q3.What is Rotary Position Embedding (RoPE)?

AA relative position encoding method that rotates query and key vectors in 2D coordinate planes using complex numbers, capturing relative distance naturally.

BA technique that rotates model weights during quantization.

CAn attention head permutation algorithm.

DA token vocabulary compressor.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Prompt Engineering & Structured Outputs

Next Lesson

Retrieval-Augmented Generation: Embeddings, Vector Databases & Semantic Search