How it works, and how anyone knows

S01 3Blue1Brown · 2024 · Video

The best short explanation of transformer mechanics in any medium: embeddings as directions in space, attention as tokens updating each other's meaning, softmax at the end. You will not be able to implement one afterward, and you will understand every subsequent conversation better.

What to do Watch it once at normal speed, on the page, without taking notes. Then rewatch only the attention section and say out loud what a single attention head does to one token. Replay that section if the explanation still feels slippery. That sentence makes the rest of the map easier to follow.

Also in Start here.

27 min YouTube · free

S02 Vaswani, Shazeer, Parmar et al. · 2017 · Paper

The transformer. Discards recurrence and convolution for self-attention, which parallelises beautifully on the hardware that existed. Everything here descends from it, and the fact that it targeted machine translation tells you nobody involved knew what they had.

What to do Read the abstract, section 3, and Figure 1, then stop. Do not attempt the training details. The one thing to leave with is the causal chain: removing recurrence made the computation parallel, which made the hardware relevant, which made scale possible. That chain, not the architecture diagram, is the paper’s actual legacy.

1 hr arXiv 1706.03762 · free

S03 Rich Sutton · 2019 · Essay

The short essay behind a thousand lab strategy arguments: methods that scale with computation keep beating hand-built human knowledge, and the field keeps finding this emotionally inconvenient. Read it before anyone tells you their clever domain-specific trick has escaped the general pattern. It probably has not. Sorry to the trick.

What to do Read it whole, twice. Then note the cleverest domain-specific trick in your own field and predict what happens to it. Date the prediction. This is the only entry on the page you should treat as a bet you have placed.

10 min incompleteideas.net · free

S04 Kaplan, McCandlish et al. · 2020 · Paper

Loss falls as a smooth power law in compute, data and parameters across seven orders of magnitude. The paper that turned capability into a budgeting exercise and justified the buildout. Its compute-optimal recommendations were later corrected; the underlying regularity was not.

What to do Read the abstract and stare at the log-log plots until the straightness stops being surprising. Note the compute-optimal recommendation specifically, because you are about to watch it get corrected. Skip the appendices entirely.

1 hr arXiv 2001.08361 · free

S05 Hoffmann et al., DeepMind · 2022 · Paper

Corrects Kaplan: for a fixed compute budget, models had been far too large and trained on far too little data, and parameters and tokens should scale together. One result that redirected billions and ended the parameter-count race.

What to do Read this immediately after Kaplan, same sitting, or the correction loses its force. Look for two things: the corrected ratio between parameters and tokens, and the fact that a result this narrow redirected billions of dollars. Then try one sentence on what changed and what did not.

45 min arXiv 2203.15556 · free

S06 Ouyang et al., OpenAI · 2022 · Paper

RLHF, and the result that made ChatGPT possible: a 1.3B model tuned on human preferences beat a 175B base model on human evaluation. The lesson taken everywhere was that the gap between a language model and a useful assistant is a fine-tuning problem, not a scale problem. Also the origin of most present-day complaints about sycophancy.

What to do Read the abstract and the human-evaluation comparison, then find the sycophancy discussion. The load-bearing fact is that 1.3B beat 175B on human preference. Afterward, notice one behaviour in a model you use daily that this stage installed rather than pretraining. That distinction is easy to miss and worth noticing.

1 hr arXiv 2203.02155 · free

S07 Bai et al., Anthropic · 2022 · Paper

RLAIF: replaces most human preference labelling with a model critiquing its own outputs against a written set of principles. Cheaper, and it makes the value specification an inspectable document rather than an emergent property of who you hired to label data.

What to do Read the method section, then go and find the actual constitution and read it. The point is not the technique. The point is that value specification became a document a stranger can read and disagree with. Try naming one specific disagreement and what you might write instead.

1 hr arXiv 2212.08073 · free

S08 Wei et al., Google · 2022 · Paper

Asking the model to show its working improves multi-step accuracy, and the effect only appears above a certain scale. A small paper with enormous consequences: it made inference-time computation a lever, which is the entire basis of the reasoning-model era.

What to do Read the abstract and the emergence figure. Extract one fact — the effect only appears above a scale threshold — then connect it forward yourself: this is why inference-time compute became a lever, and therefore why reasoning models exist at all. It is a compact paper that explains an entire era.

40 min arXiv 2201.11903 · free

S09 Yao et al. · 2022 · Paper

Interleaves reasoning traces with tool calls so a model can look things up, act, observe and revise. The architectural ancestor of every agent framework now shipping, and reading it shows how thin the actual idea is next to the marketing around it.

What to do Read the method and one worked trace. Then close the tab and sketch the loop from memory in a few lines. It makes the relatively small mechanism beneath the agent marketing easier to recognise.

40 min arXiv 2210.03629 · free

S10 Lewis et al., Meta · 2020 · Paper

Fetch relevant documents at inference time and condition generation on them rather than hoping the weights memorised it. The original is much narrower than what the acronym now means, which is worth knowing before you use the term in a design review.

What to do Read the abstract, then stop and compare the original architecture against what your team means when it says RAG. Note the difference in one line. That precise version can make later design conversations much shorter.

40 min arXiv 2005.11401 · free

S11 OpenAI · 2023 · Technical report + system card

The moment the genre changed from paper to capability dossier. It is less revealing about architecture than you will want and more revealing about the new template: benchmark sweep, limitations, red-team findings, system card, deployment posture. Annoying as a paper, essential as a historical artifact.

What to do Read it as a genre artifact, not a paper. Skim the benchmark sweep, then read the system card’s red-team findings properly. Then list what is absent: architecture, data, training compute. That absence is the historical event this document records, and it is why every later release reads the way it does.

35 min openai.com · free

S12 DeepSeek · 2025 · Paper

Reinforcement learning on verifiable outcomes producing long reasoning traces, with the recipe published openly. Technically the clearest public account of how reasoning models are trained; strategically, the moment the open-versus-closed capability gap became a live policy question rather than a theoretical one.

What to do Read the RL recipe section and one full reasoning trace. Take away two things, one technical and one political: how verifiable rewards produce long traces, and what it meant that the recipe was published at all. Hold both, because the policy track later assumes you have.

1 hr arXiv 2501.12948 · free