Advanced

18 min read

#BERT#GPT#T5#LLaMA#Encoder-Decoder#Decoder-Only#LLMs#NLP

19. Transformer Families (BERT, GPT, T5, LLaMA & Decoder-Only)

Taxonomy of transformer models: Autoencoding (BERT Masked LM), Autoregressive (GPT causal decoder), Sequence-to-Sequence (T5), and modern open LLMs.

Transformer Models: Complete Notes (Beginner to Advanced)

1. Transformer Models#

Transformer models are neural network architectures designed to process sequences using attention mechanisms rather than recurrence as their main mechanism.

The Transformer architecture was introduced in the paper "Attention Is All You Need" (2017). It became the foundation for many modern language models.

A Transformer can be organized into three broad model types:

Architecture & Data Flow
                    Transformer Models
                           |
          +----------------+----------------+
          |                |                |
     Encoder-Only     Decoder-Only    Encoder-Decoder
          |                |                |
         BERT             GPT              T5

The key difference is which parts of the Transformer architecture are used and how the model is trained.

Main Transformer Model Families#

Model family	Main component	Typical training objective	Common use
Encoder-only	Encoder	Masked language modeling	Understanding/classification
Decoder-only	Decoder	Autoregressive language modeling	Text generation
Encoder-decoder	Both	Sequence-to-sequence denoising	Translation, summarization, generation

2. BERT#

BERT stands for Bidirectional Encoder Representations from Transformers.

BERT is an encoder-only Transformer model developed by Google.

Its main purpose is to learn rich contextual representations of text.

Basic Architecture#

text
Input Text
    |
Tokenization
    |
Input Embeddings
    |
Transformer Encoder Layers
    |
Contextual Representations
    |
Task-Specific Head
    |
Output

BERT uses the Transformer encoder, which allows each token to attend to other tokens in the input sequence.

Example#

Consider:

›The animal didn't cross the street because it was too tired.

The representation of it can use information from both:

›left context  <--- it --->  right context

This bidirectional context is a major characteristic of BERT.

Important Point#

"BERT is bidirectional" does not mean that BERT generates text in both directions.

It means that during its main pretraining objective, a token can use contextual information from both sides of the sequence.

3. BERT Pretraining#

The original BERT paper used two major pretraining objectives:

Masked Language Modeling (MLM)
Next Sentence Prediction (NSP)

3.1 Masked Language Modeling#

Some input tokens are masked, and the model learns to predict them.

Example:

text
Input:
The cat sat on the [MASK].

Target:
mat

The model receives surrounding context and predicts the masked token.

Conceptually:

Architecture & Data Flow
The cat sat on the [MASK]
              |
              v
       BERT Encoder
              |
              v
        Predict "mat"

This allows BERT to learn contextual representations without requiring left-to-right generation.

3.2 Next Sentence Prediction#

BERT's original pretraining setup also included a task where the model predicted whether one sentence followed another in the original text.

Architecture & Data Flow
Sentence A + Sentence B
          |
          v
        BERT
          |
          v
Is B the actual next sentence?

Important: later research showed that NSP is not always necessary, and many BERT-style models use different pretraining objectives.

4. BERT for Downstream Tasks#

After pretraining, BERT can be adapted to different tasks.

Text Classification#

text
Text
 |
BERT
 |
[CLS] representation
 |
Classification Head
 |
Class

Examples:

Sentiment classification
Spam detection
Topic classification

Token Classification#

A prediction can be produced for each token.

Examples:

Named Entity Recognition (NER)
Part-of-speech tagging

Architecture & Data Flow
John    lives    in    Hyderabad
 |       |       |        |
Person  Other  Other    Location

Question Answering#

BERT can be used to identify the start and end positions of an answer span in a passage.

Architecture & Data Flow
Context + Question
        |
       BERT
        |
 Start position + End position

5. GPT#

GPT stands for Generative Pre-trained Transformer.

GPT models are primarily decoder-only Transformer models.

They are designed especially for autoregressive text generation.

Architecture & Data Flow
Previous Tokens
      |
      v
Decoder Transformer
      |
      v
Next-token probabilities
      |
      v
Next token
      |
      +----> fed back as context

Example#

Suppose the model receives:

›The weather today is

It predicts a probability distribution for the next token:

Architecture & Data Flow
sunny   -> 0.45
good    -> 0.20
cold    -> 0.15
...

A token is selected according to the decoding strategy, appended to the sequence, and the process continues.

6. GPT and Causal Attention#

GPT uses causal self-attention.

A token can attend to:

›itself + previous tokens

but not future tokens.

Example:

›The cat sat

When predicting the next token:

›The cat sat [next]

the model cannot see the future token.

A causal attention mask creates this restriction.

text
Token 1: can see 1
Token 2: can see 1, 2
Token 3: can see 1, 2, 3
Token 4: can see 1, 2, 3, 4

This makes the architecture suitable for autoregressive generation.

7. GPT Training Objective#

GPT is trained primarily using next-token prediction.

For a sequence:

›x1, x2, x3, ..., xT

the model learns:

text
P(x2 | x1)
P(x3 | x1, x2)
P(x4 | x1, x2, x3)
...
P(xT | x1, ..., xT-1)

The model therefore learns to estimate:

›P(x1, x2, ..., xT)

as a product of conditional probabilities:

Mathematical Formulation
P(x1, ..., xT)
=
Π P(xt | x1, ..., x(t-1))

During training, the correct previous tokens are available as context. During generation, previously generated tokens become part of the context.

8. T5#

T5 stands for Text-To-Text Transfer Transformer.

T5 is an encoder-decoder Transformer model.

Its central idea is to represent many NLP tasks as:

›text input  ->  text output

Example#

Translation:

text
Input:
translate English to German: Hello

Output:
Hallo

Summarization:

text
Input:
summarize: [long article]

Output:
[short summary]

Question answering:

text
Input:
question: What is the capital of France? context: ...

Output:
Paris

The same general model architecture can therefore be trained for many different tasks.

9. T5 Architecture#

T5 contains both an encoder and a decoder.

Architecture & Data Flow
Input Text
    |
    v
+-----------+
|  Encoder  |
+-----------+
      |
      | encoder representations
      v
+-----------+
|  Decoder  |
+-----------+
      |
      v
Output Tokens

The decoder uses:

Causal self-attention over previously generated output tokens.
Cross-attention over the encoder's representations.
Feed-forward layers.

The encoder processes the complete input sequence using non-causal self-attention.

10. Encoder-Only Models#

An encoder-only model uses the encoder portion of the Transformer architecture.

Examples:

BERT
RoBERTa
DistilBERT

The encoder generally uses bidirectional/non-causal self-attention.

Architecture & Data Flow
Input tokens
     |
     v
Encoder
     |
     v
Contextual representations
     |
     +----> Classification
     +----> Token classification
     +----> Embeddings
     +----> Extractive QA

Why Encoder-Only Models Are Good for Understanding#

Every token can generally attend to other tokens in the input.

Therefore, the representation of a word can incorporate both left and right context.

For example:

›I went to the bank to deposit money.

The representation of bank can use:

text
went to the
     +
deposit money

to understand that bank refers to a financial institution.

Typical Applications#

Text classification
Sentiment analysis
Named entity recognition
Semantic similarity
Embedding generation
Extractive question answering

Important Limitation#

Encoder-only models are not naturally designed for unrestricted left-to-right text generation because their standard attention pattern is not causal.

11. Decoder-Only Models#

A decoder-only model uses the decoder stack without the encoder-decoder cross-attention part.

Examples:

GPT-family models
Many modern large language models

Its main attention mechanism is causal self-attention.

Architecture & Data Flow
Input tokens
     |
     v
Causal Self-Attention
     |
     v
Feed-Forward Network
     |
     v
Repeated Transformer Blocks
     |
     v
Next-token probabilities

Main Strength#

Decoder-only models are naturally suited to:

Text generation
Chat
Code generation
Story generation
Completion
General language modeling

Example#

text
Prompt:
Machine learning is

Model predicts:
a

Then:

›Machine learning is a

The model predicts another token.

This continues until generation stops.

12. Encoder-Decoder Models#

An encoder-decoder model contains both Transformer components.

The encoder reads the input.

The decoder generates the output.

Architecture & Data Flow
                 Input
                   |
                   v
              +---------+
              | Encoder |
              +---------+
                   |
            Encoder states
                   |
                   v
              +---------+
 Previous --->| Decoder |----> Output token
 output       +---------+

Two Different Attention Operations in the Decoder#

1. Decoder Self-Attention

The decoder attends to previous output tokens.

Architecture & Data Flow
y1, y2, ..., y(t-1)
          |
          v
   Causal self-attention

2. Cross-Attention

The decoder attends to encoder representations.

Architecture & Data Flow
Decoder queries
      |
      v
Cross-Attention <--- Encoder keys and values
      |
      v
Decoder representation

This lets the decoder use information from the input while generating the output.

Typical Applications#

Machine translation
Summarization
Text transformation
Sequence-to-sequence generation

T5 is a major example.

13. Autoregressive Models#

An autoregressive model predicts the next part of a sequence based on previously available parts.

For text:

Architecture & Data Flow
x1 -> predict x2
x1,x2 -> predict x3
x1,x2,x3 -> predict x4
...

The probability of a sequence can be factorized as:

Mathematical Formulation
P(x1, x2, ..., xT)
=
P(x1)
P(x2 | x1)
P(x3 | x1,x2)
...
P(xT | x1,...,xT-1)

Autoregressive Generation#

Suppose the prompt is:

›Deep learning is

Generation proceeds conceptually as:

Architecture & Data Flow
Deep learning is
        |
        v
       "a"
        |
        v
Deep learning is a
        |
        v
       "field"
        |
        v
Deep learning is a field
        |
       ...

At every step, the model predicts a distribution over the next token.

Autoregressive Transformer Models#

Decoder-only Transformers are the most common architecture for autoregressive language modeling.

However, autoregressive describes a training/generation objective or factorization, not an architecture name by itself.

14. Masked Language Models#

A Masked Language Model (MLM) learns to predict tokens that have been hidden or masked from the input.

Example:

›The cat is [MASK] on the mat.

The model predicts:

›sitting

The important characteristic is that the model can use contextual information surrounding the masked position.

›Left context  ---> [MASK] <--- Right context

Training Process#

Conceptually:

Architecture & Data Flow
Original text
     |
     v
Randomly mask selected tokens
     |
     v
Masked sequence
     |
     v
Transformer Encoder
     |
     v
Predict masked tokens
     |
     v
Calculate loss
     |
     v
Update parameters

Example#

Original:

›The dog chased the ball.

Masked:

›The dog [MASK] the ball.

Target:

›chased

The model learns representations that capture relationships between words.

15. MLM vs Autoregressive Language Modeling#

These are different language-modeling objectives.

Masked Language Modeling#

Input:

›The dog [MASK] the ball.

The model predicts:

›chased

The model can use information from both sides of the masked position.

Autoregressive Language Modeling#

Input:

›The dog

Predict:

›chased

Then:

›The dog chased

Predict:

›the

Then continue.

Comparison#

Property	Masked LM	Autoregressive LM
Example	BERT	GPT
Typical architecture	Encoder-only	Decoder-only
Main objective	Predict masked tokens	Predict next token
Attention	Usually bidirectional/non-causal	Causal
Generation	Not its primary objective	Natural strength
Context for prediction	Both sides around mask	Previous tokens

16. BERT vs GPT vs T5#

Feature	BERT	GPT	T5
Architecture	Encoder-only	Decoder-only	Encoder-decoder
Main objective	Masked LM	Autoregressive LM	Text-to-text denoising/seq2seq
Attention	Bidirectional/non-causal	Causal	Encoder non-causal + decoder causal + cross-attention
Primary strength	Understanding	Generation	Input-to-output transformation
Typical tasks	Classification, NER, embeddings	Generation, chat, code	Translation, summarization, text-to-text
Generates naturally?	No	Yes	Yes

17. Encoder-Only vs Decoder-Only vs Encoder-Decoder#

Architecture & Data Flow
                 Transformer Model Types
                         |
        +----------------+----------------+
        |                |                |
        v                v                v
 Encoder-only      Decoder-only     Encoder-decoder
        |                |                |
      BERT              GPT              T5
        |                |                |
   Understand         Generate       Transform

Encoder-Only#

text
Input
  |
Encoder
  |
Representation

Best suited for extracting information from an input.

Decoder-Only#

Architecture & Data Flow
Previous tokens
      |
   Decoder
      |
Next token

Best suited for generating sequences autoregressively.

Encoder-Decoder#

Architecture & Data Flow
Input -> Encoder -> Representations
                         |
                         v
                    Decoder -> Output

Best suited for transforming one sequence into another.

18. Important Relationship Between the Concepts#

These terms describe different dimensions of Transformer models.

Architecture#

text
Encoder-only
Decoder-only
Encoder-decoder

Training Objective#

text
Masked Language Modeling
Autoregressive Language Modeling
Denoising / Sequence-to-Sequence Objectives

Model Examples#

Architecture & Data Flow
BERT  -> Encoder-only + MLM
GPT   -> Decoder-only + Autoregressive LM
T5    -> Encoder-decoder + Text-to-text/denoising objective

This distinction is important.

"Encoder-only" and "autoregressive" are not competing labels at exactly the same level.

Architecture tells us what components the model contains.

Training objective tells us what the model is trained to predict.

19. End-to-End Mental Model#

BERT#

text
Text
 |
Tokenize
 |
Mask some tokens during training
 |
Encoder
 |
Bidirectional contextual representations
 |
Predict masked tokens

Think:

"Understand the text using context from the whole input."

GPT#

text
Prompt
 |
Tokenize
 |
Causal decoder
 |
Predict next token
 |
Append token
 |
Predict next token
 |
Repeat

Think:

"Read what came before and generate what comes next."

T5#

Architecture & Data Flow
Input text
 |
Encoder
 |
Input representation
 |
Decoder + cross-attention
 |
Generate output text

Think:

"Read one piece of text and transform it into another piece of text."

20. Summary#

Concept	Simple meaning
BERT	Encoder-only Transformer designed primarily for language understanding
GPT	Decoder-only Transformer designed primarily for autoregressive generation
T5	Encoder-decoder Transformer using a text-to-text framework
Encoder-only	Uses Transformer encoder stack
Decoder-only	Uses Transformer decoder stack with causal self-attention
Encoder-decoder	Uses both encoder and decoder
Autoregressive model	Predicts sequence elements from previously available elements
Masked Language Model	Predicts hidden/masked tokens using surrounding context

21. Quick Recap#

Architecture & Data Flow
BERT
  -> Encoder-only
  -> Masked Language Modeling
  -> Bidirectional context
  -> Understanding

GPT
  -> Decoder-only
  -> Autoregressive next-token prediction
  -> Causal attention
  -> Generation

T5
  -> Encoder-decoder
  -> Text-to-text
  -> Encoder reads input
  -> Decoder generates output
  -> Cross-attention connects them

One-Line Mental Model#

Mathematical Formulation
BERT = Understand
GPT  = Generate
T5   = Transform

Knowledge Checkpoint

19. Transformer Models Taxonomy Checkpoint

Q1.What is the primary architectural difference between BERT and GPT?

ABERT is an Encoder-only model with bidirectional attention for representation learning, while GPT is a Decoder-only autoregressive model with causal (masked) attention for text generation.

BBERT uses recurrence, whereas GPT uses convolutions.

CGPT does not use self-attention.

DBERT has no feedforward layers.

Q2.What pre-training objective made BERT effective for language understanding?

AMasked Language Modeling (MLM), where 15% of tokens are masked and the model predicts them from bidirectional context, combined with Next Sentence Prediction (NSP).

BNext token prediction from left to right only.

CImage reconstruction with mean squared error.

DAdversarial training with a discriminator.

Q3.What is a major advantage of the Decoder-Only architecture (LLaMA, GPT-4, Mistral) for foundation language models?

AUnified autoregressive pre-training that directly aligns with prompt completion, in-context zero/few-shot learning, and KV-cache enabled efficient decoding.

BIt completely avoids matrix multiplications.

CIt trains with zero GPU memory.

DIt requires no token vocabulary.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

18. Transformers Architecture & Multi-Head Self-Attention

Next Lesson

20. High-Dimensional Embeddings & Vector Representations