Transformers & Large Language Models

From "Attention Is All You Need" to GPT-4o and Claude — how the transformer architecture revolutionized AI, explained so you can ace any interview AND build production systems.

Why Transformers

Before 2017, sequence modeling meant RNNs and LSTMs. They processed tokens one-by-one, left to right. Like reading a book by looking at one word at a time through a keyhole.

The RNN bottleneck:

• Sequential processing = no parallelism = slow training
• Long-range dependencies get "forgotten" (vanishing gradients)
• Hidden state is a fixed-size bottleneck — all context must squeeze through it

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph RNN["RNN: Sequential (Slow)"]
        direction LR
        W1(["Word 1"]) --> H1{{"h1"}} --> H2{{"h2"}} --> H3{{"h3"}} --> H4{{"h4"}}
        W2(["Word 2"]) --> H2
        W3(["Word 3"]) --> H3
        W4(["Word 4"]) --> H4
    end

    subgraph TR["Transformer: Parallel (Fast)"]
        direction LR
        T1(("Word 1")) ~~~ T2(("Word 2")) ~~~ T3(("Word 3")) ~~~ T4(("Word 4"))
        T1 <--> T2
        T1 <--> T3
        T1 <--> T4
        T2 <--> T3
        T2 <--> T4
        T3 <--> T4
    end

    style RNN fill:#FEF3C7,stroke:#D97706,color:#000
    style TR fill:#D1FAE5,stroke:#059669,color:#000

In June 2017, Vaswani et al. published "Attention Is All You Need". The key insight: you don't need recurrence at all. Let every token attend to every other token in parallel. Training went from days to hours. Performance leaped.

Self-Attention Mechanism

The Party Analogy

Imagine you're at a party (the input sequence). You are one word in a sentence.

• Query (Q): "What am I looking for?" — your question to the room
• Key (K): "What do I have to offer?" — everyone's name tag
• Value (V): "What information do I carry?" — the actual conversation content

You look at everyone's name tag (Q dot K), figure out who's relevant to you (softmax), then take a weighted mix of their conversations (multiply by V). That's attention.

Scaled Dot-Product Attention

Walkthrough: "The cat sat on the mat"

For the word "sat":

1. Compute Q for "sat", K and V for all words
2. Dot product Q_sat with K_the, K_cat, K_sat, K_on, K_the, K_mat
3. Highest scores likely: "cat" (subject doing the sitting) and "mat" (location)
4. Softmax normalizes these scores
5. Weighted sum of V vectors = context-aware representation of "sat"

Now "sat" knows WHO sat (cat) and WHERE (mat). Without recurrence. In one step.

Multi-Head Attention

One attention head captures one type of relationship. Multiple heads capture different patterns simultaneously.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    Input(["Input Embeddings"]) --> H1{{"Head 1<br/>Syntax"}}
    Input --> H2{{"Head 2<br/>Semantics"}}
    Input --> H3{{"Head 3<br/>Position"}}
    Input --> H4{{"Head 4<br/>Coreference"}}
    H1 --> Concat[/"Concatenate"/]
    H2 --> Concat
    H3 --> Concat
    H4 --> Concat
    Concat --> Linear[["Linear Projection"]]
    Linear --> Output(["Multi-Head Output"])

    style Input fill:#E3F2FD,stroke:#1565C0,color:#000
    style Concat fill:#FEF3C7,stroke:#D97706,color:#000
    style Output fill:#D1FAE5,stroke:#059669,color:#000
    style H1 fill:#EDE9FE,stroke:#7C3AED,color:#000
    style H2 fill:#EDE9FE,stroke:#7C3AED,color:#000
    style H3 fill:#EDE9FE,stroke:#7C3AED,color:#000
    style H4 fill:#EDE9FE,stroke:#7C3AED,color:#000

Each head has its own Q, K, V weight matrices. GPT-3 uses 96 heads. Each head has dimension d_model/n_heads. Outputs are concatenated and projected back.

Positional Encoding

Transformers process all tokens simultaneously. That's their superpower (parallelism) and their weakness (no sense of word order). "Dog bites man" and "Man bites dog" would look identical without position info.

Sinusoidal Encoding (Original)

The original paper uses sine and cosine functions of different frequencies:

Why sine/cosine? Because relative positions can be represented as linear transformations. Position 5 relative to position 3 is always the same rotation, regardless of absolute position.

Modern Approaches

Transformer Architecture

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph Encoder["Encoder Block"]
        direction LR
        IE(["Input Embedding<br/>+ Positional Encoding"]) --> MHA1{{"Multi-Head<br/>Self-Attention"}}
        MHA1 --> AN1[["Add & Norm"]]
        AN1 --> FFN1{{"Feed-Forward<br/>Network"}}
        FFN1 --> AN2[["Add & Norm"]]
    end

    subgraph Decoder["Decoder Block"]
        direction LR
        OE(["Output Embedding<br/>+ Positional Encoding"]) --> MMHA{{"Masked Multi-Head<br/>Self-Attention"}}
        MMHA --> AN3[["Add & Norm"]]
        AN3 --> CMHA{{"Cross-Attention<br/>(to Encoder)"}}
        CMHA --> AN4[["Add & Norm"]]
        AN4 --> FFN2{{"Feed-Forward<br/>Network"}}
        FFN2 --> AN5[["Add & Norm"]]
    end

    AN2 --> CMHA
    AN5 --> LIN[/"Linear + Softmax"/]
    LIN --> OUT(["Output Probabilities"])

    style Encoder fill:#E3F2FD,stroke:#1565C0,color:#000
    style Decoder fill:#EDE9FE,stroke:#7C3AED,color:#000
    style LIN fill:#FEF3C7,stroke:#D97706,color:#000
    style OUT fill:#D1FAE5,stroke:#059669,color:#000

Key Components

Residual Connections (Add): Output = Layer(x) + x. Helps gradients flow through deep networks. Without them, 96-layer models wouldn't train.

Layer Normalization (Norm): Normalizes across features (not batch). Stabilizes training. Modern models use Pre-LN (norm before attention) rather than Post-LN (norm after).

Feed-Forward Network (FFN): Two linear transformations with activation between them.

FFN(x) = max(0, xW_1 + b_1)W_2 + b_2

This is where the model stores "knowledge." Attention routes information; FFN processes it. Think: attention is the postal system, FFN is the workers at each office.

Encoder vs Decoder Models

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph ENC["Encoder-Only"]
        direction LR
        E1(["BERT, RoBERTa<br/>DeBERTa"])
        E2{{"Bidirectional<br/>Sees full context"}}
        E3[/"Classification,<br/>NER, QA"/]
    end

    subgraph DEC["Decoder-Only"]
        direction LR
        D1(["GPT, Claude<br/>Llama, Mistral"])
        D2{{"Autoregressive<br/>Left-to-right"}}
        D3[/"Text Generation,<br/>Chat, Code"/]
    end

    subgraph ED["Encoder-Decoder"]
        direction LR
        ED1(["T5, BART<br/>mBART"])
        ED2{{"Bidirectional encoder<br/>+ Autoregressive decoder"}}
        ED3[/"Translation,<br/>Summarization"/]
    end

    style ENC fill:#E3F2FD,stroke:#1565C0,color:#000
    style DEC fill:#EDE9FE,stroke:#7C3AED,color:#000
    style ED fill:#D1FAE5,stroke:#059669,color:#000

Tokenization

Models don't see words. They see tokens — sub-word units. "unhappiness" might become ["un", "happiness"] or ["un", "happ", "iness"].

Why Tokenization Matters

• Determines vocabulary size (affects model size and efficiency)
• Affects how well the model handles rare words, code, non-English text
• A bad tokenizer wastes context window on redundant tokens

Methods

Token vs Word Count

"I'm learning transformers!" 
Words:   4
Tokens:  5  ("I", "'m", " learning", " transform", "ers", "!")
         Actually 6 with the punctuation

The LLM Landscape

How LLMs Are Trained

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    PT(["Pre-Training<br/>Next Token Prediction<br/>Trillions of tokens"]) --> SFT{{"Supervised<br/>Fine-Tuning<br/>Instruction following"}}
    SFT --> RLHF{{"RLHF / DPO<br/>Human preference<br/>alignment"}}
    RLHF --> Deploy(["Deployed<br/>Model"])

    style PT fill:#E3F2FD,stroke:#1565C0,color:#000
    style SFT fill:#FEF3C7,stroke:#D97706,color:#000
    style RLHF fill:#EDE9FE,stroke:#7C3AED,color:#000
    style Deploy fill:#D1FAE5,stroke:#059669,color:#000

Stage 1: Pre-Training

• Objective: Predict the next token given all previous tokens
• Data: Trillions of tokens from the internet, books, code, papers
• Compute: Thousands of GPUs for weeks/months. GPT-4 training cost estimated at $100M+
• Result: A "base model" — great at completing text, terrible at following instructions

The base model is like a brilliant student who has read everything but has never been told what a "question" is.

Stage 2: Supervised Fine-Tuning (SFT)

• Human-written examples of ideal responses to instructions
• Typically 10K-100K high-quality examples
• Teaches the model to be helpful, follow instructions, format outputs

Stage 3: Alignment

RLHF (Reinforcement Learning from Human Feedback):

1. Generate multiple responses to a prompt
2. Humans rank responses (A > B > C)
3. Train a reward model to predict human preferences
4. Optimize the LLM policy using PPO to maximize reward

Constitutional AI (Anthropic's approach):

• Model critiques its own responses against a set of principles
• Self-improvement loop reduces need for human labelers
• Principles like "be helpful," "be harmless," "be honest"

DPO (Direct Preference Optimization):

• Skip the reward model entirely
• Directly optimize the policy from preference pairs
• Simpler, more stable, increasingly popular
• Used by Llama 3, Zephyr, and many others

Prompt Engineering

Prompting Strategies

Chain-of-Thought Example

Without CoT: "What's 17 * 24?" -> "__(often wrong)__"

With CoT: "What's 17 * 24? Think step by step."
-> "17 * 24 = 17 * 20 + 17 * 4 = 340 + 68 = 408" (correct!)

Sampling Parameters

Context Windows

The context window is the maximum number of tokens a model can process in a single forward pass. It's like the model's "working memory."

Why Context Length is Hard

Attention is O(n^2) in sequence length. Doubling context = 4x memory and compute. That's why extending context is an active research area.

Solutions for Long Context

Scaling Laws

Chinchilla Scaling (2022)

DeepMind's Chinchilla paper showed: for a given compute budget, you should scale data and model size equally.

Previous models (like GPT-3 175B) were undertrained — too many parameters, not enough data. Chinchilla (70B) matched GPT-3 performance with 4x more training data and 3x fewer parameters.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph Before["Before Chinchilla"]
        direction LR
        B1(("Bigger model = better"))
        B2[/"GPT-3: 175B params<br/>300B tokens"/]
    end

    subgraph After["After Chinchilla"]
        direction LR
        A1(("Balance params & data"))
        A2[/"Chinchilla: 70B params<br/>1.4T tokens"/]
    end

    Before --> After

    style Before fill:#FCE7F3,stroke:#DB2777,color:#000
    style After fill:#D1FAE5,stroke:#059669,color:#000

Key Scaling Insights

• Compute-optimal ratio: ~20 tokens per parameter (Chinchilla rule)
• Loss scales as power law with compute, data, and parameters
• Inference cost matters too: Llama 3 70B trained on 15T tokens (overtraining) because a smaller model with more training is cheaper to serve
• Emergent abilities: Some capabilities appear suddenly at certain scales (debated)

Inference Optimization

Training happens once. Inference happens millions of times. Optimization here directly saves money.

KV Cache

During autoregressive generation, recompute Q for only the new token but reuse K and V from all previous tokens. Without KV cache, generating N tokens is O(N^2). With it, O(N).

Problem: KV cache grows linearly with sequence length and batch size. A 70B model with 128K context needs ~40GB just for KV cache.

Speculative Decoding

Use a small "draft" model to generate N candidate tokens quickly. Then verify all N tokens with the large model in a single forward pass. If most are correct, you get N tokens for the cost of 1 large-model call.

• Draft model: 1B parameters, very fast
• Verify model: 70B parameters, accurate
• Typical speedup: 2-3x with no quality loss

Flash Attention

Standard attention materializes the full N x N attention matrix in GPU memory. Flash Attention tiles the computation to keep everything in fast SRAM. Result: 2-4x speedup, less memory, exact same output. Not an approximation — pure IO optimization.

Quantization

Reduce model precision from float16 to int8 or int4. Smaller model = faster inference + less memory.

Serving Frameworks

Open vs Closed Models

When to Self-Host

• Regulated industries (healthcare, finance): data can't leave your infrastructure
• High volume: >$5K/month in API costs often means self-hosting is cheaper
• Low latency: need <100ms time-to-first-token
• Custom models: heavily fine-tuned for your domain
• Offline environments: edge deployment, air-gapped systems

When to Use APIs

• Prototyping: get started in minutes, not days
• Low volume: don't justify GPU infrastructure
• Frontier quality: need the absolute best model available
• Multi-modal: vision + audio + code in one API
• Small team: no ML ops expertise in-house

Interview Questions