Think of an LLM like an extraordinarily well-read librarian who has memorized billions of texts. The result can feel like understanding, but the process is fundamentally different.
1. Introduction: The Illusion of Understanding
Fluency can feel like understanding, though the mechanism underneath is statistical pattern-matching.
2. Tokenization: Breaking Down Language
Text is split into tokens, the units a model can actually process.
2.1 What Is a Token?
The minimal unit of processing: a word, a subword fragment, a character, whitespace, punctuation or a control marker.
2.2 Tokenization Algorithms
The main algorithms that decide where text is split.
Byte-Pair Encoding (BPE)
Merges the most frequent pair of symbols, again and again, until the vocabulary reaches its target size.
WordPiece
Merges the pair that most increases the likelihood of the corpus, marking subwords inside a word with a prefix.
SentencePiece
Works on raw byte streams without assuming spaces, so it is language-agnostic and robust to formatting.
Score(A, B) = P(“AB”) / (P(“A”) × P(“B”))
3. Embeddings: The Language of Vector Space
Token IDs become dense vectors that capture relationships between meanings.
3.1 Why Integers Aren’t Enough
An ID number carries no relationship to other IDs, so the model needs a richer representation.
3.2 The Embedding Matrix
A learned lookup table maps every token to its vector.
3.3 Semantic Geometry
Vector arithmetic reflects relationships that emerge from training.
3.4 Positional Encoding: Giving Tokens a Sense of Order
Attention alone ignores word order, so position information is added.
embedding(“king”) − embedding(“man”) + embedding(“woman”) ≈ embedding(“queen”)
4. The Transformer Architecture
A stack of identical blocks that refine the representation of a sequence.
4.1 The Big Picture
One block: attention, normalization, residual connections and a feed-forward network.
4.2 Three Architectural Variants
Encoder-only, decoder-only and encoder-decoder designs, and what each is best at.
4.3 The Feed-Forward Network (FFN)
A per-position network that expands, applies a non-linearity, and compresses back.
4.4 Layer Normalization
Keeps activations numerically stable during training.
LayerNorm(x) = γ × (x − μ) / (σ + ε) + β
5. The Attention Mechanism
Each token selectively draws information from the other tokens in its context.
5.1 The Core Idea
Like citing sources in research: look up what is relevant, ignore the rest.
5.2 Scaled Dot-Product Attention
Queries, keys and values, and the formula that combines them.
5.3 Causal Masking: Preventing “Cheating”
During generation a token may not see positions that come after it.
5.4 Multi-Head Attention: Parallel Specialization
Several attention heads run in parallel and their outputs are combined.
Attention(Q, K, V) = softmax( QKᵀ / √d_k ) × V
6. Next Token Prediction
The final hidden state becomes probabilities over the vocabulary, and one token is chosen.
6.1 From Hidden State to Vocabulary Scores
The last representation is projected into logits over the vocabulary.
6.2 Converting Logits to Probabilities
Softmax turns logits into a probability distribution.
6.3 Decoding Strategies
Greedy decoding, beam search, temperature sampling and top-p.
P(token_i | context) = exp(logit_i) / Σ_j exp(logit_j)
7. Autoregressive Generation
Text is generated one token at a time until a stop condition.
7.1 The Generation Loop
From the prompt, through sampling, to the stop token.
7.2 The KV-Cache: Avoiding Redundant Work
Caching key and value vectors avoids recomputing them at every step.
8. Training: How an LLM Learns
The next-token objective, data at scale, and the optimization loop.
8.1 The Training Objective
Predict the next token at every position.
8.2 Training Data at Scale
How many tokens models such as GPT-3 and LLaMA were trained on.
8.3 The Optimization Loop
Forward pass, backward pass, and parameter updates with AdamW.
L = −(1/T) × Σₜ log P(tokenₜ | token₁…ₜ₋₁)
9. Why It Works: Emergence and Scale
Scaling laws, and capabilities that appear as models grow.
9.1 Scaling Laws
Loss falls predictably as parameters, data and compute increase.
9.2 Emergent Capabilities
Abilities that appear around certain model sizes.
10. Known Limitations and Failure Modes
Hallucinations, reasoning weaknesses, long-context recall and knowledge cutoffs.
10.1 Hallucinations
Confident false output, often triggered by obscure facts or exact figures.
10.2 Weak Causal and Mathematical Reasoning
Novel multi-step maths, state tracking and counterfactuals remain hard.
10.3 The “Lost in the Middle” Problem
Information in the middle of a long context is often poorly recalled.
10.4 Knowledge Cutoff
A model knows nothing after its training data ends.
11. Further Reading
- Attention Is All You Need
- BERT
- Language Models are Few-Shot Learners (GPT-3)
- Scaling Laws for Neural Language Models
- Training Compute-Optimal Large Language Models (Chinchilla)
- LLaMA: Open and Efficient Foundation Language Models
- Direct Preference Optimization
- FlashAttention
- The Illustrated Transformer
- Andrej Karpathy: Let’s build GPT from scratch
- 3Blue1Brown: But what is a GPT?