From Numbers to Words: How LLMs Turn Tokens into Text

1. Introduction: The Illusion of Understanding

When we type a question into a language model and receive a coherent, sophisticated response, it’s tempting to believe the system “understands” language the way a human does. But this intuition hides a mechanism that is radically different from human cognition.

Analogy: Think of an LLM like an extraordinarily well-read librarian who has memorized billions of texts. When you ask a question, they don’t reason from first principles — they pattern-match your query against everything they’ve read and construct the most statistically plausible continuation. The result can feel like understanding, but the process is fundamentally different.

An LLM does not:

• Form causal mental models of the environment
• Have personal experiences or memories between conversations
• Access any ground truth external to its training data

What it does do — extraordinarily well — is learn statistical patterns over sequences of text at a scale that results in behaviors that look like understanding.

By the end of this guide, you will understand:

1. How raw text becomes mathematical representations a computer can manipulate
2. How those representations flow through a neural network to produce predictions
3. Why a conceptually simple mechanism, repeated billions of times, produces surprising emergent behaviors
4. Where the real limits of this technology lie

2. Tokenization: Breaking Down Language

2.1 What Is a Token?

Before an LLM can process any text, it must break that text into tokens — the minimal units of processing the model recognizes.

Analogy: Tokens are to an LLM what Lego bricks are to a builder. You don’t work with raw plastic — you work with standardized pieces. Some bricks represent full words; others represent just a syllable or a punctuation mark.

Token Type	Example
A complete word	"cat", "run"
A subword fragment	"transform" + "ation"
A single character	"a", "1", "!"
Whitespace	" ", " "
Punctuation	",", ".", "?"
A special control marker	[BOS], [EOS], [PAD]

2.2 Tokenization Algorithms

Three algorithms dominate modern LLMs:

Byte-Pair Encoding (BPE)

Used by: GPT-2, GPT-3, GPT-4, Llama 1

BPE works by iteratively merging the most frequent character pairs in a corpus:

Step 1:  vocabulary = all individual characters
Step 2:  find the most frequent adjacent pair  →  e.g. ("t", "h") → "th"
Step 3:  merge it into a single new token
Step 4:  repeat until vocabulary reaches target size (e.g. 50,000 tokens)

Result: Frequent words become single tokens; rare words decompose into familiar fragments.

WordPiece

Used by: BERT, RoBERTa, DistilBERT

Similar to BPE, but merges the pair that maximizes the likelihood of the corpus, not just raw frequency:

Score(A, B) = P("AB") / (P("A") × P("B"))

Subwords that appear mid-word are prefixed with ##:

"tokenization"  →  "token", "##iza", "##tion"

SentencePiece

Used by: LLaMA 2 & 3, T5, Mistral, multilingual models

Key difference: treats the raw byte stream as input, without assuming spaces delimit words. This makes it:

• Language-agnostic (works equally well for Chinese, Arabic, English)
• Robust to inconsistent spacing or formatting

3. Embeddings: The Language of Vector Space

3.1 Why Integers Aren’t Enough

Token IDs like 412 and 413 have no inherent meaning relative to each other. The model needs a richer representation — one that captures semantic relationships between words.

The solution: convert each token ID into a dense vector — a list of real numbers in a high-dimensional space.

Analogy: Imagine a map where every word is a dot. Words with similar meanings are placed close together; unrelated words are far apart. Embeddings are the coordinates of each word on that map — but in hundreds or thousands of dimensions instead of two.

3.2 The Embedding Matrix

At the start of training, the model initializes an embedding matrix E of shape [V × d]:

• V = vocabulary size (e.g., 50,000)
• d = embedding dimension (e.g., 768 for BERT-base, 4,096 for LLaMA-7B)

Looking up a token’s embedding is simply selecting its row from E:

embedding("cat") = E[8203] = [ 0.23, -0.87, 1.12, 0.04, ..., -0.55 ]

3.3 Semantic Geometry

The most fascinating property of learned embeddings: arithmetic on vectors corresponds to semantic relationships in language.

embedding("king") - embedding("man") + embedding("woman") ≈ embedding("queen")
embedding("Paris") - embedding("France") + embedding("Germany") ≈ embedding("Berlin")

Nobody programmed these relationships. They emerge from training on text alone.

3.4 Positional Encoding: Giving Tokens a Sense of Order

The attention mechanism processes all tokens simultaneously — it has no built-in notion of order. Positional encodings solve this by adding a position-specific vector to each token’s embedding.

Method	Used By	Key Property
Sinusoidal (fixed)	Original Transformer	Simple, no extra parameters
RoPE (rotary)	LLaMA, Mistral, GPT-NeoX	Generalizes to longer sequences
ALiBi (linear bias)	MPT, BLOOM	Penalizes attention over long distances

4. The Transformer Architecture

4.1 The Big Picture

The Transformer is a stack of identical blocks, each refining the representation of the sequence.

┌─────────────────────────────────────────────┐
│               TRANSFORMER BLOCK             │
│                                             │
│   Input                                     │
│     │                                       │
│     ▼                                       │
│  LayerNorm                                  │
│     │                                       │
│     ▼                                       │
│  Multi-Head Self-Attention                  │
│     │                                       │
│     ▼                                       │
│  + Residual (skip connection)               │
│     │                                       │
│     ▼                                       │
│  LayerNorm                                  │
│     │                                       │
│     ▼                                       │
│  Feed-Forward Network (FFN)                 │
│     │                                       │
│     ▼                                       │
│  + Residual (skip connection)               │
│     │                                       │
│     ▼                                       │
│   Output (feeds into next block)            │
└─────────────────────────────────────────────┘

4.2 Three Architectural Variants

Variant	Examples	Best For
Encoder-Only	BERT, RoBERTa	Classification, NER, sentence similarity
Decoder-Only	GPT-4, LLaMA, Mistral	Open-ended text generation, chat
Encoder-Decoder	T5, BART	Translation, summarization, QA

4.3 The Feed-Forward Network (FFN)

Each Transformer block contains an FFN applied independently at every token position. Its structure:

Input (d dimensions)
      ↓
  Linear layer  →  expand to 4×d dimensions
      ↓
  Non-linearity (GELU or SwiGLU)
      ↓
  Linear layer  →  compress back to d dimensions
      ↓
Output (d dimensions)

4.4 Layer Normalization

Applied before each sublayer to keep activations numerically stable during training:

LayerNorm(x) = γ × (x - μ) / (σ + ε) + β

5. The Attention Mechanism

5.1 The Core Idea

Analogy: Imagine you’re writing a research paper and you need to cite sources. For each sentence you write, you don’t re-read your entire library — you selectively pull from the sources most relevant to that specific sentence. The attention mechanism does exactly this: for each token, it learns to selectively “pull” information from the other tokens most relevant to understanding it in this context.

5.2 Scaled Dot-Product Attention

Every token in the sequence is projected into three roles:

Vector	Meaning	Question it answers
Query (Q)	What I’m looking for	“What context do I need?”
Key (K)	What I advertise	“What can I offer others?”
Value (V)	What I actually contain	“What information do I carry?”

Attention(Q, K, V) = softmax( QK^T / √d_k ) × V

5.3 Causal Masking: Preventing “Cheating”

In generation models, a token must not be allowed to peek at future tokens. The causal mask enforces this:

Before softmax, set score(i, j) = -∞  for all j > i

5.4 Multi-Head Attention: Parallel Specialization

Rather than a single attention mechanism, Transformers run h attention heads in parallel, each with its own independent Q/K/V projections:

MultiHead output = Concat(head_1, ..., head_h) × W_O

6. Next Token Prediction

6.1 From Hidden State to Vocabulary Scores

After all Transformer layers, every token position has a rich contextual representation vector of dimension d. This vector is projected into vocabulary space by a final linear layer (the LM Head):

logits = h_last × W_lm      (shape: [V])

6.2 Converting Logits to Probabilities

Softmax converts raw logits into a proper probability distribution:

P(token_i | context) = exp(logit_i) / Σ_j exp(logit_j)

6.3 Decoding Strategies

Strategy	Description	Use Case
Greedy Decoding	Always picks the highest probability token	Fast, deterministic, but repetitive
Beam Search	Maintains k candidate sequences	Translation, summarization
Temperature Sampling	Scales logits by 1/T	Controls randomness
Top-p (Nucleus)	Samples from smallest set with cumulative prob ≥ p	Balanced creativity

7. Autoregressive Generation

7.1 The Generation Loop

┌────────────────────────────────────────────────────────┐
│                  GENERATION LOOP                       │
│                                                        │
│  1. Start with the user's prompt as token sequence     │
│  2. Run the full Transformer forward pass              │
│  3. Read the output at the last position               │
│  4. Apply softmax → probability distribution           │
│  5. Sample next token using chosen strategy            │
│  6. Append new token to the sequence                   │
│  7. If token == [EOS] or max_length reached → STOP     │
└────────────────────────────────────────────────────────┘

7.2 The KV-Cache: Avoiding Redundant Work

The KV-Cache stores the Key and Value vectors for all previously processed tokens:

	Without KV-Cache	With KV-Cache
Compute per step	O(n²)	O(n)
Speed	Slow	Much faster

8. Training: How an LLM Learns

8.1 The Training Objective

Pre-training has one deceptively simple goal: predict the next token at every position in every training document.

L = - (1/T) × Σ_{t=1}^{T}  log P(token_t | token_{1..t-1})

8.2 Training Data at Scale

Model	Training Tokens	Approx. Pages of Text
GPT-3	300B	~240 billion pages
LLaMA-1	1.4T	~1.1 trillion pages
LLaMA-3	15T	~12 trillion pages

8.3 The Optimization Loop

Training adjusts billions of parameters using backpropagation + the AdamW optimizer:

For each batch of text:
  1. FORWARD PASS
  2. BACKWARD PASS
  3. PARAMETER UPDATE (AdamW)

9. Why It Works: Emergence and Scale

9.1 Scaling Laws

LLM performance follows predictable power laws as a function of model size, data, and compute:

Loss(N) ∝ N^(-α_N)     ← more parameters → lower loss
Loss(D) ∝ D^(-α_D)     ← more data → lower loss
Loss(C) ∝ C^(-α_C)     ← more compute → lower loss

9.2 Emergent Capabilities

Capability	Appears at approximately
Simple arithmetic	1B parameters
Multi-step reasoning	10B parameters
Chain-of-thought prompting	50–100B parameters
Calibrated uncertainty	50B+ parameters

10. Known Limitations and Failure Modes

10.1 Hallucinations

LLMs sometimes generate confident-sounding falsehoods. Common triggers:

• Questions about obscure people, places, or events
• Requests for exact numbers, dates, or verbatim quotes
• Topics where plausible-sounding wrong answers exist

10.2 Weak Causal and Mathematical Reasoning

LLMs are very good at pattern-matching reasoning they’ve seen before. They struggle with:

• Novel multi-step math problems
• Problems requiring careful tracking of intermediate state
• True counterfactual reasoning

10.3 The “Lost in the Middle” Problem

Even with large context windows, LLMs don’t use all positions equally. Information near the start and end is best recalled. Information buried in the middle is often “forgotten.”

10.4 Knowledge Cutoff

Models have no awareness of events after their training data ends. Workarounds include RAG, tool use, and frequent retraining.

11. Further Reading

Foundational Papers

• Attention Is All You Need (Vaswani et al., 2017) — arxiv.org/abs/1706.03762
• BERT (Devlin et al., 2018) — arxiv.org/abs/1810.04805
• Language Models are Few-Shot Learners / GPT-3 (Brown et al., 2020) — arxiv.org/abs/2005.14165
• Scaling Laws for Neural Language Models (Kaplan et al., 2020) — arxiv.org/abs/2001.08361
• Training Compute-Optimal Large Language Models / Chinchilla (Hoffmann et al., 2022) — arxiv.org/abs/2203.15556
• LLaMA: Open and Efficient Foundation Language Models (Touvron et al., 2023) — arxiv.org/abs/2302.13971
• Direct Preference Optimization (Rafailov et al., 2023) — arxiv.org/abs/2305.18290
• Flash Attention (Dao et al., 2022) — arxiv.org/abs/2205.14135

Accessible Explanations

• The Illustrated Transformer — Jay Alammar’s visual walkthrough — jalammar.github.io/illustrated-transformer
• Andrej Karpathy — Let’s build GPT from scratch — youtube.com/watch?v=kCc8FmEb1nY
• 3Blue1Brown — But what is a GPT? — youtube.com/watch?v=wjZofJX0v4M

12. Technical Glossary

Term	Definition
Token	Minimal unit of text processed by the model.
Embedding	Vector of real numbers representing a token in high-dimensional space.
Transformer	Neural network architecture based on attention mechanisms.
Self-Attention	Mechanism where each token computes its relationship to every other token.
Logits	Unnormalized scores produced by the final model layer.
Softmax	Function converting logits into a probability distribution.
Temperature	Divisor applied to logits before softmax; controls randomness.
KV-Cache	Stores Key/Value vectors to avoid recomputation during generation.
Pre-training	Training phase on large text corpora to predict the next token.
RLHF	Reinforcement Learning from Human Feedback.
Context Window	Maximum number of tokens the model can process simultaneously.
Hallucination	Generation of factually incorrect information.
Scaling Laws	Empirical power-law relationships between size, data, compute, and loss.
RAG	Retrieval Augmented Generation.
Quantization	Reduces weight precision to cut memory and inference cost.