A 45-minute walkthrough of the architecture that powers GPT, Claude, Gemini, LLaMA, Mistral — every modern LLM.
45 min lecture · 15 min Q&A · Working takeaway repo to keep.
Challengers ramping into AI-first engineering. Interrupt anytime.
One architecture. Almost every modern AI system.
GPT-4, Claude, Gemini, LLaMA, Mistral. All transformers.
Vision Transformers (ViT) match or beat CNNs on most benchmarks.
Image, audio, video — same architecture, different tokenizers.
Cursor, Claude Code, agentic systems — all transformer-based.
How do we get a neural network to understand order-dependent data?
Variable length. Sentences, paragraphs, books — input size isn't fixed.
Long-range dependencies. "The girl who lived in the house with the red door is sad" — sad refers to girl, 11 tokens back.
Order matters. "Dog bites man" ≠ "Man bites dog".
Ambiguity. "Bank" means river-side or financial — needs context.
Process tokens one at a time, maintaining a hidden state.
At each step, read one token + the previous hidden state, produce a new hidden state. The hidden state is supposed to be a running summary of everything seen so far.
LSTMs and GRUs improved on this with gating mechanisms, but the fundamental shape stayed the same: sequential, state-passing.
These motivated the entire transformer architecture.
You can't compute step t+1 until step t is done. Modern GPUs love parallelism — RNNs refuse to give it to them.
By the time you've processed 1,000 tokens, the hidden state has been overwritten 1,000 times. Information from early tokens fades.
Let every token look at every other token, directly, in one step.
Each token asks: "which other tokens are most relevant to me?"
what I'm looking for
what I offer / what describes me
what I contribute if you pick me
Three linear projections, one matrix multiplication, one softmax. That's it.
Three learned linear layers. X is (n × d), Q/K/V are (n × d).
Q = X·Wq · K = X·Wk · V = X·WvResult is (n × n): one score for every (token, token) pair.
scores = Q · KᵀDividing by √d keeps gradients stable. Softmax turns scores into a probability distribution.
weights = softmax(scores / √d)Each token's new representation is a blend of all values, weighted by relevance.
output = weights · VAn n × n grid where cell (i, j) is how much token i attends to token j.
Softmax over keys. Every query distributes its attention across all available keys.
Causal mask — during training a token can't attend to future tokens (would be cheating).
These weights are learned. The model decides what to attend to.
Run attention many times in parallel, each on a different slice of the embedding.
With h heads, the model simultaneously tracks:
Attention is permutation-invariant. "Dog bites man" and "Man bites dog" look identical to it.
Shuffle the input tokens — same attention weights, same output. That's not what we want for language.
Now "cat" at position 3 is a different vector than "cat" at position 7.
Fixed cos/sin functions of position. No parameters. Extends to longer sequences than seen in training.
PE(pos, 2i) = sin(pos / 10000^(2i/d))
Just an embedding table indexed by position. Simple, but capped at max training length.
PE = nn.Embedding(max_len, d)
Rotate Q and K vectors by position-dependent angles. Encodes relative position naturally.
applied in attention, not at embedding
Attention is the core. Stack it with a feed-forward network, residuals, and layer-norm — that's a block.
Lets each token gather information from every other token. This is where context comes in.
A two-layer MLP applied per-position. Where most parameters live; does the heavy lifting on each token's representation.
Add the input back to the output of each sublayer. Lets gradients flow through deep stacks without vanishing.
Normalize activations across the embedding dimension. Stabilizes training, lets us go deep.
These two tricks are why we can stack 100+ transformer blocks and still train them.
For each token vector, subtract its mean, divide by its standard deviation. Then a learned scale (γ) and shift (β) per dimension.
Very different capabilities.
Every token attends to every other token. Great for understanding tasks — classification, NER, embedding generation.
Each token only attends to previous tokens. Trained on next-token prediction. The architecture behind every modern chatbot.
Encoder reads input bidirectionally; decoder generates output causally, attending to the encoded input via cross-attention. Built for seq-to-seq.
One simple objective, applied to the internet. That's it.
Training data is free. Every sentence on the internet is a training example. No human labels needed — the next word IS the label.
To predict the next word well, you need to model grammar, facts, reasoning, style, code, math, dialogue. Compression of human knowledge.
Models don't see characters or words. They see tokens — chunks chosen by a learned algorithm.
It's not just that they're more accurate. They train an order of magnitude faster.
Each step waits for the previous one.
Latency: O(n) where n = sequence length.
Computed in one matrix multiplication.
Latency: O(1) wall-clock — assuming enough GPU.
Once you can parallelize across the sequence dim, you can throw orders-of-magnitude more compute at the problem. That's what unlocked the entire scaling-laws era.
Caveat: attention is O(n²) in memory, which is its own bottleneck and the main focus of long-context research today.
Test loss falls predictably as a power law in parameters, data, and compute.
Bigger model — more layers, wider FFN, more heads.
More tokens to train on.
More FLOPs (= N × D × constant).
vs. the 2017 paper.
One mental model to carry away.
Tokenizer turns text into integers; embedding table turns integers into vectors. Position info is added.
Q, K, V projection → softmax(QKᵀ/√d) · V. Every token gets to look at every other token.
Two-layer MLP applied per-position. This is where most of the model's capacity lives.
Residual connections + layer norm let you go deep. 12 blocks (BERT-base) to 100+ (GPT-4 class).
Final linear projects back to vocab. Cross-entropy loss. Trained on trillions of tokens. That's a modern LLM.
Everything we just covered, in ~250 lines of PyTorch. Trains on a laptop.
Loss drops from ~4.2 to ~1.5 in a few minutes on CPU. Then the model produces text like:
ROMEO: Thou shalt not fear, the morning of my love Hath made the heavens to weep, and the world To mourn the loss of thy companion...
The original paper. Surprisingly readable. Start here.
Best visual explanation on the internet. Free.
2-hour video. Builds essentially this repo, live-coded. Excellent.
The original paper, line-by-line, with PyTorch.
Production-quality cousin of the takeaway repo. Reproduces GPT-2.
If you want to understand what's happening inside the weights.