01 · about a week
Here is the genuinely wild part: the thing driving the entire global conversation is, mechanically, a next-token predictor with extremely expensive furniture around it. These are the papers that built it, in the order they landed. You do not need the maths. You need to know which idea came from where, so the jargon stops being jargon.
The primary literature in the order the ideas arrived, because the vocabulary is historical and reading it out of sequence makes it look arbitrary. You do not need the maths. You need to know which paper introduced which idea, so when someone says scaling laws or RLHF you know what they are pointing at and what it did and did not establish.
The visual explanation goes first because reading transformer papers cold is a beautiful way to become sleepy and wrong. Build the mental cartoon, then let the formal version put shoes on it.
The best short explanation of transformer mechanics in any medium: embeddings as directions in space, attention as tokens updating each other's meaning, softmax at the end. You will not be able to implement one afterward, and you will understand every subsequent conversation better.
Also in Start here. Ticking it here marks it there.
27 min YouTube · free
The transformer. Discards recurrence and convolution for self-attention, which parallelises beautifully on the hardware that existed. Everything here descends from it, and the fact that it targeted machine translation tells you nobody involved knew what they had.
1 hr arXiv 1706.03762 · free
Ready with the full reading map.
Aggregate usage diagnostics are stored; your question and answer text are not.
Basically just magic
This prompt is fixed, silly, and completely uneditable. Press the button and the AI tries to produce a playable browser game with a tiny right-side leaderboard below.
pong now. rectangle friendships. ball doink. browser. make playable. no explaining.Logged-in readers only. The button appears after you sign in.
Scaling laws made the field look legible, and Chinchilla corrected the recipe. This is the part where the lab strategy stops being personality and starts being arithmetic with a burn rate.
The short essay behind a thousand lab strategy arguments: methods that scale with computation keep beating hand-built human knowledge, and the field keeps finding this emotionally inconvenient. Read it before anyone tells you their clever domain-specific trick has escaped the general pattern. It probably has not. Sorry to the trick.
10 min incompleteideas.net · free
Loss falls as a smooth power law in compute, data and parameters across seven orders of magnitude. The paper that turned capability into a budgeting exercise and justified the buildout. Its compute-optimal recommendations were later corrected; the underlying regularity was not.
1 hr arXiv 2001.08361 · free
Corrects Kaplan: for a fixed compute budget, models had been far too large and trained on far too little data, and parameters and tokens should scale together. One result that redirected billions and ended the parameter-count race.
45 min arXiv 2203.15556 · free
Pretraining gives you a powerful completion engine. The next few ideas are about turning that engine into something people can ask for help without immediately receiving the haunted autocomplete version of help.
RLHF, and the result that made ChatGPT possible: a 1.3B model tuned on human preferences beat a 175B base model on human evaluation. The lesson taken everywhere was that the gap between a language model and a useful assistant is a fine-tuning problem, not a scale problem. Also the origin of most present-day complaints about sycophancy.
1 hr arXiv 2203.02155 · free
RLAIF: replaces most human preference labelling with a model critiquing its own outputs against a written set of principles. Cheaper, and it makes the value specification an inspectable document rather than an emergent property of who you hired to label data.
1 hr arXiv 2212.08073 · free
Asking the model to show its working improves multi-step accuracy, and the effect only appears above a certain scale. A small paper with enormous consequences: it made inference-time computation a lever, which is the entire basis of the reasoning-model era.
40 min arXiv 2201.11903 · free
Interleaves reasoning traces with tool calls so a model can look things up, act, observe and revise. The architectural ancestor of every agent framework now shipping, and reading it shows how thin the actual idea is next to the marketing around it.
40 min arXiv 2210.03629 · free
Fetch relevant documents at inference time and condition generation on them rather than hoping the weights memorised it. The original is much narrower than what the acronym now means, which is worth knowing before you use the term in a design review.
40 min arXiv 2005.11401 · free
After the classic papers, read the modern compression. DeepSeek shows what a current technical report sounds like; Karpathy shows the whole kitchen without pretending the spoon is magic.
The moment the genre changed from paper to capability dossier. It is less revealing about architecture than you will want and more revealing about the new template: benchmark sweep, limitations, red-team findings, system card, deployment posture. Annoying as a paper, essential as a historical artifact.
35 min openai.com · free
Reinforcement learning on verifiable outcomes producing long reasoning traces, with the recipe published openly. Technically the clearest public account of how reasoning models are trained; strategically, the moment the open-versus-closed capability gap became a live policy question rather than a theoretical one.
1 hr arXiv 2501.12948 · free
The entire training stack in one sitting: crawling and filtering, tokenisation, pretraining, supervised fine-tuning, RLHF. Shows what each stage installs and what it cannot, explains hallucination structurally, and explains why the system has no persistent self between conversations. No position on risk anywhere in it.
Also in The risk argument. Ticking it here marks it there.
3.5 hr YouTube · free