Think of an LLM like an extraordinarily well-read librarian who has memorized billions of texts. The result can feel like understanding, but the process is fundamentally different.

1. Introduction: The Illusion of Understanding

Fluency can feel like understanding, though the mechanism underneath is statistical pattern-matching.

2. Tokenization: Breaking Down Language

Text is split into tokens, the units a model can actually process.

2.1 What Is a Token?

The minimal unit of processing: a word, a subword fragment, a character, whitespace, punctuation or a control marker.

2.2 Tokenization Algorithms

The main algorithms that decide where text is split.

Byte-Pair Encoding (BPE)

Merges the most frequent pair of symbols, again and again, until the vocabulary reaches its target size.

WordPiece

Merges the pair that most increases the likelihood of the corpus, marking subwords inside a word with a prefix.

SentencePiece

Works on raw byte streams without assuming spaces, so it is language-agnostic and robust to formatting.

Score(A, B) = P(“AB”) / (P(“A”) × P(“B”))

3. Embeddings: The Language of Vector Space

Token IDs become dense vectors that capture relationships between meanings.

3.1 Why Integers Aren’t Enough

An ID number carries no relationship to other IDs, so the model needs a richer representation.

3.2 The Embedding Matrix

A learned lookup table maps every token to its vector.

3.3 Semantic Geometry

Vector arithmetic reflects relationships that emerge from training.

3.4 Positional Encoding: Giving Tokens a Sense of Order

Attention alone ignores word order, so position information is added.

embedding(“king”) − embedding(“man”) + embedding(“woman”) ≈ embedding(“queen”)

4. The Transformer Architecture

A stack of identical blocks that refine the representation of a sequence.

4.1 The Big Picture

One block: attention, normalization, residual connections and a feed-forward network.

4.2 Three Architectural Variants

Encoder-only, decoder-only and encoder-decoder designs, and what each is best at.

4.3 The Feed-Forward Network (FFN)

A per-position network that expands, applies a non-linearity, and compresses back.

4.4 Layer Normalization

Keeps activations numerically stable during training.

LayerNorm(x) = γ × (x − μ) / (σ + ε) + β

5. The Attention Mechanism

Each token selectively draws information from the other tokens in its context.

5.1 The Core Idea

Like citing sources in research: look up what is relevant, ignore the rest.

5.2 Scaled Dot-Product Attention

Queries, keys and values, and the formula that combines them.

5.3 Causal Masking: Preventing “Cheating”

During generation a token may not see positions that come after it.

5.4 Multi-Head Attention: Parallel Specialization

Several attention heads run in parallel and their outputs are combined.

Attention(Q, K, V) = softmax( QKᵀ / √d_k ) × V

6. Next Token Prediction

The final hidden state becomes probabilities over the vocabulary, and one token is chosen.

6.1 From Hidden State to Vocabulary Scores

The last representation is projected into logits over the vocabulary.

6.2 Converting Logits to Probabilities

Softmax turns logits into a probability distribution.

6.3 Decoding Strategies

Greedy decoding, beam search, temperature sampling and top-p.

P(token_i | context) = exp(logit_i) / Σ_j exp(logit_j)

7. Autoregressive Generation

Text is generated one token at a time until a stop condition.

7.1 The Generation Loop

From the prompt, through sampling, to the stop token.

7.2 The KV-Cache: Avoiding Redundant Work

Caching key and value vectors avoids recomputing them at every step.

8. Training: How an LLM Learns

The next-token objective, data at scale, and the optimization loop.

8.1 The Training Objective

Predict the next token at every position.

8.2 Training Data at Scale

How many tokens models such as GPT-3 and LLaMA were trained on.

8.3 The Optimization Loop

Forward pass, backward pass, and parameter updates with AdamW.

L = −(1/T) × Σₜ log P(tokenₜ | token₁…ₜ₋₁)

9. Why It Works: Emergence and Scale

Scaling laws, and capabilities that appear as models grow.

9.1 Scaling Laws

Loss falls predictably as parameters, data and compute increase.

9.2 Emergent Capabilities

Abilities that appear around certain model sizes.

10. Known Limitations and Failure Modes

Hallucinations, reasoning weaknesses, long-context recall and knowledge cutoffs.

10.1 Hallucinations

Confident false output, often triggered by obscure facts or exact figures.

10.2 Weak Causal and Mathematical Reasoning

Novel multi-step maths, state tracking and counterfactuals remain hard.

10.3 The “Lost in the Middle” Problem

Information in the middle of a long context is often poorly recalled.

10.4 Knowledge Cutoff

A model knows nothing after its training data ends.

11. Further Reading

  1. Attention Is All You Need
  2. BERT
  3. Language Models are Few-Shot Learners (GPT-3)
  4. Scaling Laws for Neural Language Models
  5. Training Compute-Optimal Large Language Models (Chinchilla)
  6. LLaMA: Open and Efficient Foundation Language Models
  7. Direct Preference Optimization
  8. FlashAttention
  9. The Illustrated Transformer
  10. Andrej Karpathy: Let’s build GPT from scratch
  11. 3Blue1Brown: But what is a GPT?