LLM mechanisms 🧠
Not how to prompt a model — how one works. This track goes from a string of text to the integers a network multiplies, builds a GPT one component at a time, trains and fine-tunes it, then takes it apart again to find out which parts do what.
Every module states the essence of its subject in plain language, why it matters, the full list of topics it covers, and runnable Python for each idea. The code is the point: nearly every claim here is something you can check in a notebook in under a minute.
Where to start
- New to deep learning? Start at Part 8, then Part 7, then come back to Part 1. Parts 7 and 8 are last in the ordering but first in the dependency graph.
- Comfortable with PyTorch? Start at Part 1 and read straight through. Parts 1–3 build the model and the tooling; Parts 4–6 take it apart.
- Here for interpretability? Read Part 2's “Build a GPT” first regardless — Parts 5 and 6 refer to specific tensors inside that architecture on every page.
Part 1
Tokenization and embeddings
How text becomes numbers, and how those numbers become a geometry where meaning is a direction.
Words to tokens to numbers
How text becomes integers a network can multiply — and why that conversion causes a surprising share of LLM failures.
19 topics · 16 min
Embedding spaces
How token ids become geometry — static vs. contextual embeddings, cosine similarity, position embeddings, and training embeddings yourself.
17 topics · 18 min
Part 2
Large language models
Build a GPT from an embedding table upwards, then pretrain, fine-tune and instruction-tune it.
Build a GPT
Assemble a working GPT from an embedding table upwards — softmax and sampling, layernorm, causal attention, the transformer block, multi-head attention, and the GPU.
28 topics · 24 min
Pretrain LLMs
Training a model from random weights — AdamW, custom losses, weight initialisation, dropout, numerical scaling and the datasets involved.
19 topics · 19 min
Fine-tune pretrained models
Adapting an existing model to your data or task — full and partial fine-tuning, freezing, PEFT and LoRA, classification heads, clipping and schedulers.
21 topics · 21 min
Instruction tuning and RLHF
How a text predictor becomes an assistant — chat templates, instruction datasets, and reinforcement learning from human feedback.
6 topics · 14 min
Part 3
Evaluating LLMs
The numbers — perplexity, benchmarks, divergences — and the structured looking that finds what numbers miss.
Quantitative evaluation
Measuring a model with numbers — perplexity, masked-prediction accuracy, HellaSwag, KL divergence, MAUVE, benchmark suites and bias assessment.
14 topics · 20 min
Qualitative evaluation
Looking at the model instead of scoring it — black-box evaluation, red-teaming, activation distributions and token heatmaps.
6 topics · 12 min
Part 4
AI safety and interpretability
Why alignment is a specification problem, and why understanding the mechanism is the only way to verify a fix.
AI safety and alignment
Why 'just make it safe' is not a specification — in-context learning, scaling, a hands-on prompt attack, and how to work on this seriously.
6 topics · 14 min
Mechanistic interpretability
What mech interp is, how it relates to safety, its vocabulary, its theoretical and empirical styles, and the criticisms worth taking seriously.
5 topics · 13 min
Part 5
Observational mech interp
Measuring what a model represents — embeddings, neurons, layers, trajectories and circuits — without changing anything.
Investigating token embeddings
Measuring the geometry of embedding space — cosine similarity at scale, t-SNE and DBSCAN, RSA, semantic axes, kNN, graphs and the SVD spectrum.
20 topics · 22 min
Investigating neurons and dimensions
Finding what an individual unit responds to — activation maximisation, PyTorch hooks, multi-token words, and statistical tests for tuning.
17 topics · 22 min
Investigating layers
Depth-resolved analysis — Q/K/V similarity structure, RSA by layer, effective dimensionality, mutual information and the logit lens.
13 topics · 20 min
Embedding trajectories through the model
Treating a token's representation as a path — rotation angles, path length, residual-stream decomposition and parts-of-speech comparisons.
7 topics · 15 min
Identifying circuits and components
From individual units to mechanisms — attention heads, sparse probing, latent vs. manifest variables, sparse autoencoders and generalised eigendecomposition.
13 topics · 21 min
Part 6
Causal mech interp
Editing activations to establish what the model actually uses: ablation, patching, silencing and subspace removal.
How to modify activations
The mechanics of causal interpretability — hooks that write rather than read, and the four intervention types with their different confounds.
3 topics · 14 min
Editing hidden states
Intervening on the residual stream — scaling early layers, injecting noise, correcting bias, activation patching and skipping layers entirely.
6 topics · 16 min
Interfering with attention
Head-level causal experiments — ablation, silencing, per-head patching and what each reveals about attention's role.
6 topics · 15 min
Modifying the MLP
Interventions on per-position computation — median replacement, statistically targeted lesions and removing whole subspaces.
4 topics · 13 min
Part 7
Python foundations
The slice of Python this work uses heavily, plus Matplotlib, text handling and PyTorch tensors.
Python foundations
Colab and notebooks, data types, indexing and slicing, functions, NumPy, flow control and comprehensions — the subset this work actually uses.
20 topics · 18 min
Visualisation, text processing and PyTorch
Matplotlib for figures you can read, string and text handling, and the PyTorch objects — classes, tensors, shapes and random numbers.
11 topics · 17 min
Part 8
Deep learning foundations
The maths that everything above is made of, gradient descent, and a network built from scratch.
The maths of deep learning
Transpose, dot products, matrix multiplication, softmax, logarithms, entropy, argmax, mean and variance, sampling, the t-test and derivatives.
15 topics · 22 min
How models learn: gradient descent
The optimisation algorithm behind all of it — one and two dimensions, local minima, and fixed versus adaptive learning rates.
5 topics · 14 min
The essence of deep learning modelling
The perceptron and ANN architecture, a geometric reading, the three-part maths of forward propagation, loss and backpropagation — then both in PyTorch.
7 topics · 16 min
The topic outline follows the syllabus of a course on LLM mechanisms; the explanations, code and commentary here are written for this site. Quizzes for this track are not built yet — for graded practice, see the question banks.