Advanced

26 min read

#LLMs#Pre-training#Tokenization#KV Cache#Causal Attention#Inference#Transformers

29. Large Language Models (LLM) Pre-training & Inference

End-to-end foundation model engineering: autoregressive pre-training datasets, byte-pair tokenization, causal self-attention, KV caching optimization, and decoding strategies (top-p, temperature).

Large Language Models: Complete Notes (Beginner to Advanced)

Introduction#

Large Language Models (LLMs) are neural networks trained on very large collections of text to learn patterns in language.

Modern LLMs are commonly based on the Transformer architecture.

A simplified LLM lifecycle is:

text
Large Text Dataset
       ↓
   Tokenization
       ↓
   Token IDs
       ↓
   Pretraining
       ↓
Base Language Model
       ↓
Instruction Tuning / SFT
       ↓
Aligned / Instruction-Following Model
       ↓
Inference
       ↓
Generated Text

During generation:

text
Prompt
  ↓
Tokenize
  ↓
Context
  ↓
LLM
  ↓
Next-Token Probabilities
  ↓
Sampling / Selection
  ↓
Next Token
  ↓
Repeat
  ↓
Generated Response

1. Large Language Models (LLMs)

1.1 What is an LLM?#

A Large Language Model is a neural-network model trained on large amounts of text to learn statistical patterns and representations of language.

An LLM can learn relationships involving:

Words
Tokens
Syntax
Semantics
Long-range dependencies
Facts present in its training data
Patterns of reasoning and problem solving

The term large generally refers to the scale of the model, training data, and computation, although there is no single universal parameter count that defines an LLM.

1.2 Why Are LLMs Called Language Models?#

A language model assigns probabilities to sequences of tokens.

For a sequence:

›x1, x2, x3, ..., xT

the model can represent:

›P(x1, x2, ..., xT)

Using the chain rule:

Mathematical Formulation
P(x1, ..., xT)
=
P(x1)
× P(x2 | x1)
× P(x3 | x1, x2)
× ...
× P(xT | x1, ..., xT-1)

An autoregressive LLM learns to estimate the probability of the next token given the previous context.

1.3 Transformer-Based LLMs#

Many modern LLMs use Transformer architectures.

A simplified decoder-only Transformer looks like:

text
Input Tokens
     ↓
Token Embeddings
     +
Positional Information
     ↓
Transformer Block
     ↓
Transformer Block
     ↓
Transformer Block
     ↓
...
     ↓
Final Hidden States
     ↓
Language Modeling Head
     ↓
Logits
     ↓
Probability Distribution

Each Transformer block commonly contains:

text
Causal Self-Attention
        ↓
Feed-Forward Network

with residual connections and normalization around these components.

2. Tokenization

2.1 What is Tokenization?#

Tokenization is the process of converting text into smaller units called tokens.

Example:

›"I love machine learning"

may become something conceptually like:

›["I", " love", " machine", " learning"]

The exact tokens depend on the tokenizer.

The tokenizer then maps tokens to integer IDs:

text
["I", " love", " machine", " learning"]
                ↓
[40, 912, 3812, 9274]

The exact IDs are vocabulary-specific.

2.2 Why Tokenization is Necessary#

Neural networks operate on numerical representations.

Therefore:

text
Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embeddings
 ↓
Neural Network

2.3 Subword Tokenization#

Modern LLM tokenizers often use subword-based approaches.

A word may be represented as:

›"unbelievable"

→

text
"un"
"believ"
"able"

The exact segmentation depends on the tokenizer.

Subword tokenization helps balance:

text
Vocabulary Size
        vs
Ability to Represent Unseen / Rare Words

2.4 Special Tokens#

Tokenizers may contain special tokens used for model control or representation.

Examples include:

text
BOS → Beginning of Sequence
EOS → End of Sequence
PAD → Padding
UNK → Unknown

Not every model uses all of these, and modern tokenizers can use model-specific special tokens.

3. Vocabulary

3.1 What is a Vocabulary?#

A vocabulary is the collection of tokens that a tokenizer knows.

Example:

text
Token       ID
----------------
hello       10
world       11
machine     12
learning    13

The vocabulary maps:

›Token ↔ Integer ID

3.2 Vocabulary Size#

If a tokenizer has:

›50,000 tokens

then:

Mathematical Formulation
Vocabulary Size = 50,000

For a language model, the output layer commonly produces one logit for each vocabulary token.

If:

Mathematical Formulation
Vocabulary = V

then for one position:

Mathematical Formulation
Logits shape = [V]

For a batch and sequence:

Mathematical Formulation
Logits shape = [Batch, Sequence Length, V]

3.3 Vocabulary and Embeddings#

A token ID is used to look up an embedding.

Conceptually:

text
Token ID
   ↓
Embedding Matrix
   ↓
Vector

If:

Mathematical Formulation
Vocabulary Size = V
Embedding Dimension = D

then the token embedding matrix has approximately:

›V × D

parameters.

4. Context Window

4.1 What is a Context Window?#

The context window is the maximum amount of token context a model can process at once under a particular model/configuration.

Conceptually:

text
┌──────────────────────────────────────────┐
│              Context Window              │
│                                          │
│  Token 1 ... Token 2 ... Token N        │
│                                          │
└──────────────────────────────────────────┘

The exact context length depends on the model.

4.2 Context Window During Inference#

Suppose the model supports:

›8,192 tokens

The available context includes the tokens supplied to the model and, depending on the API/model interface, generated tokens that must fit within the model's context limit.

Therefore, if a conversation becomes too long, systems may need to:

text
Truncate old context
Summarize context
Retrieve relevant context

4.3 Context Window vs Memory#

The context window is not the same thing as permanent memory.

text
Context Window
→ Tokens available to the model for a particular computation

Persistent Memory
→ Information stored outside that immediate model context

An LLM does not automatically retain every previous conversation inside every future inference request.

5. Next-Token Prediction

5.1 What is Next-Token Prediction?#

Autoregressive LLMs are commonly trained to predict the next token from previous tokens.

Example:

›"The sun rises in the"

Possible next-token probabilities:

text
east     → 0.70
morning  → 0.10
sky      → 0.05
...

The model produces a probability distribution over the vocabulary.

5.2 Mathematical Form#

Given:

›x1, x2, ..., xt

the model predicts:

›P(x(t+1) | x1, x2, ..., xt)

The predicted distribution contains one probability for every vocabulary token.

5.3 Autoregressive Generation#

Suppose the prompt is:

›"The cat"

The model predicts:

›sat

Now the sequence becomes:

›"The cat sat"

The model predicts again:

›"on"

Then:

›"The cat sat on"

The process continues:

text
Prompt
 ↓
Predict next token
 ↓
Append token
 ↓
Predict next token
 ↓
Append token
 ↓
...

5.4 Logits and Softmax#

The model first produces logits.

text
Logits
  ↓
Softmax
  ↓
Probabilities

For vocabulary size V:

Mathematical Formulation
z = [z1, z2, ..., zV]

Softmax:

Mathematical Formulation
P(i) = exp(zi) / Σj exp(zj)

The probabilities sum to approximately:

›1

Sampling methods such as temperature, top-k, and top-p can modify this distribution before selecting the next token.

6. Pretraining

6.1 What is Pretraining?#

Pretraining is the initial large-scale training stage in which an LLM learns general language representations and patterns from a large corpus.

For an autoregressive language model, a common objective is next-token prediction.

text
Large Text Corpus
       ↓
Tokenization
       ↓
Token Sequences
       ↓
Next-Token Prediction
       ↓
Loss
       ↓
Backpropagation
       ↓
Parameter Updates

6.2 Training Objective#

Given a sequence:

›x1, x2, ..., xT

the model learns to predict:

text
x2 from x1
x3 from x1,x2
...
xT from x1,...,x(T-1)

A common loss is negative log-likelihood:

Mathematical Formulation
L
=
- Σt log P(x_t | x_<t)

Often this is averaged over prediction positions and training examples.

6.3 Teacher Forcing#

During autoregressive pretraining, the model is commonly trained using the actual previous tokens as context.

For:

›"The cat sat on the mat"

the training examples conceptually include:

text
Input:  "The"
Target: "cat"

Input:  "The cat"
Target: "sat"

Input:  "The cat sat"
Target: "on"

This is often described as teacher forcing.

6.4 Pretraining Scale#

Pretraining can involve:

text
Large datasets
+
Large model
+
Large compute budget
+
Many optimization steps

The goal is to learn general-purpose representations and language modeling behavior before task-specific adaptation.

7. Instruction Tuning

7.1 What is Instruction Tuning?#

Instruction tuning trains a pretrained model to better follow natural-language instructions.

Instead of only learning:

›Predict the next token

from broad text, the model is additionally trained on examples such as:

text
Instruction:
"Summarize this paragraph."

Input:
[paragraph]

Desired Response:
[summary]

7.2 Why Instruction Tuning?#

A base language model may be good at continuing text but may not consistently behave like an assistant.

Instruction tuning teaches patterns such as:

text
Instruction
    ↓
Understand task
    ↓
Produce useful response

7.3 Instruction-Tuning Example#

text
User:
"Translate 'hello' into Japanese."

Assistant:
"こんにちは"

Another:

text
User:
"Explain overfitting in simple terms."

Assistant:
"Overfitting happens when..."

The model learns to associate instructions with desired response behavior.

8. Supervised Fine-Tuning (SFT)

8.1 What is SFT?#

Supervised Fine-Tuning (SFT) is a training stage where a pretrained model is trained on labeled input-output examples.

For an instruction-following model:

text
Instruction + Input
        ↓
Pretrained Model
        ↓
Target Response

The model's output is compared against the desired response using a supervised loss.

8.2 SFT Dataset#

An SFT dataset can contain examples such as:

json
{
    "instruction": "Explain photosynthesis.",
    "response": "Photosynthesis is..."
}

For conversational models:

text
User:
"How does a neural network learn?"

Assistant:
"It learns by..."

8.3 SFT Loss#

For target tokens:

›y1, y2, ..., yT

the model can minimize:

Mathematical Formulation
L_SFT
=
- Σt log P(y_t | prompt, y_<t)

The loss is generally computed on target response tokens according to the training setup.

8.4 Instruction Tuning vs SFT#

These terms are closely related but not perfectly interchangeable.

text
Instruction Tuning
→ Goal: make a pretrained model better at following instructions

SFT
→ Training method: learn from supervised input-output examples

Instruction tuning is often implemented using SFT.

9. RLHF

9.1 What is RLHF?#

RLHF (Reinforcement Learning from Human Feedback) is a family of alignment methods that uses human preference information to optimize model behavior.

A classic RLHF pipeline is:

text
Pretrained Model
       ↓
Supervised Fine-Tuning
       ↓
Instruction-Following Model
       ↓
Human Preference Data
       ↓
Reward Model
       ↓
Reinforcement Learning
       ↓
Aligned Model

9.2 Human Preference Data#

Humans may compare multiple responses:

text
Prompt
 ├── Response A
 └── Response B

A human evaluator may indicate:

›A is better than B

Many such comparisons can be used to train a reward model.

9.3 Reward Model#

A reward model learns to predict human preferences.

Conceptually:

text
Prompt + Response
       ↓
Reward Model
       ↓
Reward Score

If human preferences generally favor one response over another, the reward model learns to assign higher scores to responses that better match those preferences.

9.4 Reinforcement Learning Stage#

A policy model generates responses.

text
Prompt
 ↓
Policy Model
 ↓
Response
 ↓
Reward Model
 ↓
Reward
 ↓
RL Optimization
 ↓
Updated Policy

A classic implementation used PPO (Proximal Policy Optimization).

9.5 RLHF Advantages#

Uses human preference information
Can improve helpfulness and instruction following
Can optimize behavior that is difficult to specify with ordinary supervised labels

9.6 RLHF Limitations#

Human preference collection can be expensive
Reward models can be imperfect
RL training is more complex than ordinary supervised fine-tuning
Optimization can introduce undesirable behavior if the reward objective is poorly specified

10. DPO

10.1 What is DPO?#

DPO (Direct Preference Optimization) is a preference-optimization method that learns directly from preference pairs without requiring the traditional separate reward-model-plus-RL pipeline.

A preference dataset may contain:

text
Prompt
Chosen Response
Rejected Response

Example:

text
Prompt:
"Explain recursion."

Chosen:
"Recursion is a technique where..."

Rejected:
"Recursion is..."

The model is trained to prefer the chosen response.

10.2 DPO Concept#

Traditional RLHF:

text
Preference Data
      ↓
Reward Model
      ↓
Reinforcement Learning
      ↓
Policy

DPO:

text
Preference Data
      ↓
Direct Preference Optimization
      ↓
Policy

DPO uses a reference model and optimizes a classification-like objective derived from a preference-based formulation.

10.3 DPO Objective#

A common DPO objective can be written conceptually as:

Architecture & Data Flow
L_DPO
=
- log σ(
    β [
        log πθ(y_w | x) - log πref(y_w | x)
        -
        log πθ(y_l | x) + log πref(y_l | x)
    ]
)

where:

Mathematical Formulation
x   = prompt
y_w = preferred / chosen response
y_l = rejected response
πθ  = trainable policy
πref = reference policy
β    = preference-strength / temperature-like coefficient
σ    = sigmoid

The objective encourages the trainable model to assign relatively higher probability to the preferred response than the rejected response, while comparing it against a reference model.

10.4 RLHF vs DPO#

Feature	Classic RLHF	DPO
Preference data	Yes	Yes
Separate reward model	Typically yes	No
Reinforcement-learning optimization	Yes	No traditional RL stage
Training complexity	Higher	Usually simpler
Core idea	Optimize policy using learned reward	Directly optimize preferences

DPO is not the only alternative to RLHF, and real alignment pipelines can combine multiple techniques.

11. Inference

11.1 What is Inference?#

Inference is the process of using a trained model to generate predictions or outputs for new inputs.

For an LLM:

text
Prompt
  ↓
Tokenization
  ↓
Model Forward Pass
  ↓
Logits
  ↓
Token Selection / Sampling
  ↓
Next Token

The process repeats until a stopping condition is reached.

11.2 Autoregressive Inference#

Suppose:

Mathematical Formulation
Prompt = "The weather is"

The model generates:

text
"The weather is"
        ↓
"nice"
        ↓
"The weather is nice"
        ↓
"today"
        ↓
"The weather is nice today"

At every generation step:

text
Current Context
      ↓
Model
      ↓
Next-Token Distribution
      ↓
Select / Sample Token
      ↓
Append Token

11.3 Inference vs Training#

Training	Inference
Learns model parameters	Uses fixed model parameters
Requires gradients	Usually no gradients
Uses optimizer	No optimizer update
Computationally expensive	Usually cheaper per example
Produces updated model	Produces predictions/text

12. Temperature

12.1 What is Temperature?#

Temperature controls how sharply or randomly the next-token probability distribution is sampled.

Given logits:

›z

temperature scaling commonly uses:

Mathematical Formulation
P(i)
=
softmax(z_i / T)

where:

Mathematical Formulation
T = temperature

12.2 Low Temperature#

A lower temperature makes the distribution sharper.

text
Low T
 ↓
High-probability tokens become more dominant
 ↓
More deterministic behavior

Example:

text
Token A → 0.90
Token B → 0.07
Token C → 0.03

12.3 High Temperature#

A higher temperature makes the distribution flatter.

text
High T
 ↓
Lower-probability tokens receive relatively more probability
 ↓
More varied / random outputs

Conceptually:

text
Low Temperature
→ focused / predictable

High Temperature
→ diverse / less predictable

Temperature does not guarantee that every output will be deterministic or creative; it modifies the sampling distribution.

13. Top-K Sampling

13.1 What is Top-K Sampling?#

Top-K sampling limits the candidate tokens to the K tokens with the highest probabilities.

Suppose the model produces:

text
Token A → 0.40
Token B → 0.25
Token C → 0.15
Token D → 0.10
Token E → 0.05
Token F → 0.05

If:

Mathematical Formulation
K = 3

keep:

text
A
B
C

and remove the rest from the sampling candidate set.

The remaining probabilities are then renormalized.

13.2 Top-K Flow#

text
Logits
  ↓
Probability Distribution
  ↓
Select K Highest-Probability Tokens
  ↓
Remove Other Tokens
  ↓
Renormalize
  ↓
Sample

13.3 Effect of K#

Small K:

›Fewer choices
→ More focused output

Large K:

›More choices
→ More diversity

The useful range depends on the model and task.

14. Top-P Sampling

14.1 What is Top-P?#

Top-P sampling, also called nucleus sampling, selects the smallest set of highest-probability tokens whose cumulative probability reaches at least P.

Suppose:

text
A → 0.50
B → 0.25
C → 0.15
D → 0.05
E → 0.05

For:

Mathematical Formulation
P = 0.80

we accumulate:

Mathematical Formulation
A = 0.50
A+B = 0.75
A+B+C = 0.90

Therefore the nucleus contains:

›A, B, C

The probabilities are then renormalized and sampled.

14.2 Top-P Flow#

text
Probability Distribution
        ↓
Sort Tokens by Probability
        ↓
Accumulate Probability
        ↓
Keep Tokens Until Cumulative Probability ≥ P
        ↓
Renormalize
        ↓
Sample

14.3 Effect of P#

Lower P:

›Smaller candidate set
→ More focused

Higher P:

›Larger candidate set
→ More diverse

15. Temperature vs Top-K vs Top-P

These are different controls.

Method	How it Controls Sampling
Temperature	Reshapes the probability distribution
Top-K	Keeps a fixed number of highest-probability tokens
Top-P	Keeps the smallest probability mass covering a chosen cumulative probability

Conceptually:

text
Logits
  ↓
Temperature Scaling
  ↓
Top-K / Top-P Filtering
  ↓
Renormalization
  ↓
Sampling
  ↓
Next Token

The exact order and availability of these controls depends on the inference implementation.

16. Greedy Decoding

Although not one of the requested sampling methods, it is useful for understanding them.

Greedy decoding always selects the highest-probability token:

Mathematical Formulation
next_token = argmax(probabilities)

Example:

text
A → 0.60
B → 0.25
C → 0.15

Greedy decoding chooses:

›A

No random sampling is required.

17. Sampling Example

A simplified implementation:


🐍 PythonInteractive WebAssembly
import torch


def sample_next_token(
    logits,
    temperature=1.0,
    top_k=None,
    top_p=None
):
    # Temperature
    logits = logits / temperature

    # Top-K
    if top_k is not None:
        values, _ = torch.topk(logits, top_k)
        threshold = values[..., -1, None]

        logits = torch.where(
            logits < threshold,
            torch.full_like(
                logits,
                float("-inf")
            ),
            logits
        )

    # Convert to probabilities
    probabilities = torch.softmax(
        logits,
        dim=-1
    )

    # Sample
    next_token = torch.multinomial(
        probabilities,
        num_samples=1
    )

    return next_token

This is a simplified illustration. Production generation systems usually implement more efficient and feature-rich decoding logic.

18. End-to-End LLM Lifecycle

text
                 Large Text Corpus
                        │
                        ▼
                   Tokenization
                        │
                        ▼
                    Pretraining
                        │
                        ▼
                Base Language Model
                        │
                        ▼
              Instruction Tuning / SFT
                        │
                        ▼
             Instruction-Following Model
                        │
                ┌───────┴───────┐
                │               │
               RLHF            DPO
                │               │
                └───────┬───────┘
                        ▼
                  Aligned Model
                        │
                        ▼
                    Inference
                        │
                        ▼
                 Token Probabilities
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
       Temperature     Top-K      Top-P
             │          │          │
             └──────────┼──────────┘
                        ▼
                 Selected Token
                        │
                        ▼
                 Repeat Generation

The exact training pipeline differs between models. Not every modern LLM uses the same sequence of post-training methods.

19. Important Distinctions

Pretraining vs Instruction Tuning#

text
Pretraining
→ Learn general language patterns

Instruction Tuning
→ Improve instruction-following behavior

Instruction Tuning vs SFT#

text
Instruction Tuning
→ Goal / adaptation stage

SFT
→ Supervised training method commonly used to perform it

RLHF vs DPO#

text
RLHF
→ Preference data
→ Reward model
→ Reinforcement learning

DPO
→ Preference data
→ Direct preference optimization

Context Window vs Vocabulary#

text
Vocabulary
→ What token types the tokenizer/model can represent

Context Window
→ How many tokens can be considered in one model context

Temperature vs Top-K vs Top-P#

text
Temperature
→ Reshapes probabilities

Top-K
→ Fixed number of candidates

Top-P
→ Variable number of candidates based on probability mass

20. Summary Table

Concept	Core Idea
LLM	Large neural language model trained on extensive text data
Tokenization	Convert text into tokens and token IDs
Vocabulary	Collection of tokens known by the tokenizer/model
Context Window	Maximum supported token context for a model/configuration
Next-Token Prediction	Predict the next token from previous context
Pretraining	Learn general language behavior from large-scale data
Instruction Tuning	Improve following of natural-language instructions
SFT	Train on supervised input-output examples
RLHF	Use human preferences through reward modeling and RL
DPO	Directly optimize preference pairs without a traditional reward-model RL stage
Inference	Generate outputs using trained model parameters
Temperature	Controls distribution sharpness during sampling
Top-K	Restricts sampling to K highest-probability tokens
Top-P	Restricts sampling to a probability-mass nucleus

21. Quick Recap

text
TEXT
 ↓
TOKENIZATION
 ↓
TOKEN IDs
 ↓
PRETRAINING
 ↓
BASE LLM
 ↓
INSTRUCTION TUNING / SFT
 ↓
PREFERENCE ALIGNMENT
 ├── RLHF
 └── DPO
 ↓
INFERENCE
 ↓
LOGITS
 ↓
TEMPERATURE / TOP-K / TOP-P
 ↓
NEXT TOKEN
 ↓
APPEND TOKEN
 ↓
REPEAT
 ↓
FINAL RESPONSE

The core mental model is:

text
LLM
→ Predicts probabilities over the next token

Tokenization
→ Converts text ↔ token IDs

Vocabulary
→ Defines the available token set

Context Window
→ Limits how much token context the model can use

Pretraining
→ Teaches general language patterns

SFT / Instruction Tuning
→ Teaches the model to follow instructions

RLHF / DPO
→ Uses preference information to shape behavior

Inference
→ Uses the trained model to generate text

Temperature
→ Changes probability sharpness

Top-K
→ Keeps K candidate tokens

Top-P
→ Keeps the smallest candidate set covering P probability mass

Knowledge Checkpoint

29. Large Language Models Checkpoint

Q1.How does KV Caching (Key-Value Cache) accelerate autoregressive LLM token generation?

AIt stores previously computed Key and Value projection tensors across attention layers, avoiding redundant re-computation for all prompt tokens at each new token step.

BIt caches generated answers in a Redis database.

CIt skips the feedforward network layer for generated tokens.

DIt pre-computes all future output words in advance.

Q2.What is the role of Temperature and Top-p (Nucleus) sampling during LLM text decoding?

ATemperature scales logit sharpness (T < 1 makes outputs deterministic, T > 1 encourages diversity); Top-p restricts sampling to the smallest candidate token set whose cumulative probability >= p.

BTemperature controls GPU fan speed; Top-p sets vocabulary size.

CThey eliminate the need for Softmax.

DThey translate tokens into different human languages.

Q3.What is Rotary Position Embedding (RoPE) and why is it standard in modern LLMs (LLaMA, Mistral)?

AIt encodes relative token positions by rotating Query and Key vectors in complex 2D subspaces, naturally decaying attention scores with distance and generalizing across context lengths.

BIt computes token embeddings using 3D spherical rotations.

CIt removes positional encodings entirely.

DIt rotates model weight matrices during training.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

28. Deep Learning Deployment, ONNX & Inference Optimization

Next Lesson

30. Multimodal Deep Learning & Vision-Language Models