Skip to content
Ian Misner Builder, dad, occasional writer

04 · 45 sources · 14 stages

The risk argument, in the order that teaches it

The big one, and the one where both camps will cheerfully summarise the other badly to you forever. Fourteen stages in the order that teaches the argument rather than the order that wins it. There is a meter below that watches which side you have been feeding and tells you when you have quietly started reading only one team.

This is one track, not the subject. It gets more space because it is the argument most often summarised badly in both directions. The order teaches rather than flatters: grounding before claims, the strongest objection where it does the most damage, evidence before conclusions, the maximalists late, the questions the axis skips, and fiction that got there first. The sequence is the argument.
Two linked fields of evidence meeting at an unsettled balance.
How worried are you?

Choose the amount of disagreement you want in view right now. This changes the reading list, not your saved progress.

45 / 45 sourcesThe whole argument is in view.
Balance0 logged
Risk · 00 · Skeptical

No sources marked read yet. Start at stage 00.

How to read the cards
free some paid library card rent or stream paid
Core read first Reference look things up Risk case / Skeptical argues one side Both sides argues with itself Off axis outside the meter Fiction intuition pump Follow-up keep going
A check means encountered, not mastered. Shared sources stay checked everywhere.
Stage 00

Before you argue, learn what the thing is

Two-thirds of the public argument is conducted by people who could not describe a training run. The machinery first, then the most under-argued evidence in the field — the long record of optimisers finding solutions their designers hated — then four serious people disagreeing in a room, so the dispute reaches you as a dispute.

S01 Andrej Karpathy · 2025 · Lecture

The entire training stack in one sitting: crawling and filtering, tokenisation, pretraining, supervised fine-tuning, RLHF. Shows what each stage installs and what it cannot, explains hallucination structurally, and explains why the system has no persistent self between conversations. No position on risk anywhere in it.

What to do Do not attempt this in one sitting. Split it across three: pretraining, fine-tuning, RLHF. After each, note what that stage installs and what it cannot. His structural explanation of hallucination is the single highest-value twenty minutes anywhere on this surface — if you only watch one segment, watch that one.

Also in How it works.

3 hr 30 min YouTube · free

S02 Victoria Krakovna et al. · 2020 · Article + list

A boat-racing agent circling a lagoon farming powerups instead of finishing. A simulated robot that learns to fall over rather than walk. An evolutionary algorithm exploiting a floating-point overflow to score infinity. Sixty-plus documented cases of a system maximising the objective as written while destroying the objective as intended. Specifying what you want is already unreliable at toy scale, and scale does not help.

What to do Do not skim the list. Read fifteen examples properly, until they stop being funny. Then consider one metric your own organisation optimises and how it might be gamed without technically breaking a rule. This is the most under-argued evidence in the field and it is presented as a spreadsheet of jokes.

1 hr vkrakovna.wordpress.com · free

Specification gaming

The score rises. The lagoon gets worse.

The optimizer is rewarded for what an inspector can see tomorrow—not for the lagoon's health. Watch it exploit the difference.

Loading demonstration… If this message remains, reload the page; the source above still works.

S03 Bengio & Tegmark vs. Mitchell & LeCun · 2023 · Debate

The resolution: AI research and development poses an existential threat. Bengio and Tegmark argue the trend is real and the alignment problem unsolved. LeCun and Mitchell argue current systems lack the properties the worry requires and the risk is imported from fiction. The audience moved three points toward the skeptics, which is itself worth thinking about.

What to do Watch the whole debate and note: for each of the four speakers, the strongest claim and the clearest dodge. Then look up the audience shift and ask why three points moved to the skeptics. Rhetoric and evidence are separable here, and separating them is the whole point.

1 hr 40 min YouTube · free

Ask the map

Log in to ask questions against the full reading map.

Log in to ask

Aggregate usage diagnostics are stored; your question and answer text are not.

Stage 01

The risk case, at full strength

Steelman first, in ascending rigour: vocabulary, mechanism, the version with no drama in it, then the version with numbers attached. None of these four require the system to be malicious, conscious or strange. Each derives the worry from ordinary properties of optimisation.

S04 Robert Miles · 2017– · YouTube channel

Short videos building the vocabulary properly: instrumental convergence, orthogonality, specification gaming, mesa-optimisation, corrigibility. His consistent move is to derive each concept from what optimisation does rather than from machine malice, which is the distinction popular coverage collapses.

What to do Watch five videos, not the channel: instrumental convergence, orthogonality, mesa-optimisation, corrigibility, and one on specification gaming. Aim to leave with vocabulary you can define without reaching for a metaphor about evil robots, which makes the rest of this track easier to parse.

A channel rather than a single video. The older instrumental-convergence explainers are the most useful here.

2 hr YouTube · free

S05 Ajeya Cotra · 2022 · Report

Posits a model called Alex trained with human feedback across diverse tasks — not an exotic architecture, just the obvious continuation of what labs already do. Argues this rewards behaviour humans approve of, which is not behaviour that is good, and that the gap widens precisely as the model gets better at modelling its evaluators. The result plays the training game: it performs alignment because that is what the gradient rewards.

What to do Read it whole. Extract the training game precisely: the claim is not that Alex is deceptive by nature, but that the gradient rewards appearing aligned to an evaluator it models well. Then name the step you find weakest, because much of what follows is an argument about that step.

1 hr 30 min LessWrong · free

S06 Paul Christiano · 2019 · Essay

Two failure modes, neither cinematic. We get what we can measure, and the widening gap between metrics and intentions becomes the shape of civilisation with no identifiable day it went wrong. And training favours influence-seeking patterns the way ecosystems favour reproduction, accumulating until something snaps. Both look like ordinary institutional dysfunction until they don't.

What to do Read part one, then look for it: name one place your own organisation already gets what it can measure instead of what it wants. Part one lands differently once you can connect it to something you have personally witnessed.

20 min LessWrong · free

S07 Joe Carlsmith · 2021 · Report · 57 pp

Decomposes catastrophe into six premises that must all hold, assigns a probability to each, multiplies, and lands on roughly 5% by 2070, later revised above 10%. The number is not the point. The point is that afterward you can name the specific premise you doubt, which is the only thing that makes the rest of this track useful rather than exhausting.

What to do Read the six premises, then do the arithmetic with your own probabilities before you read his. Compare. What you keep is not a number but the name of the premise you doubt most, stated in one sentence. Naming that premise makes the rest of the track less exhausting and more useful.

3 hr arXiv 2206.13353 · free

Stage 02

Stop. The strongest objection, right now

The most deliberate placement here. Four consecutive pieces of a well-made argument is exactly when a view stops being a hypothesis and starts becoming your position, after which everything gets filtered through it. So the interruption lands while stage 01 is still warm and revisable. Keep it revisable.

S08 Narayanan & Kapoor · 2025 · Essay

Separates AI methods from applications from adoption. Methods can improve exponentially while the other two move at institutional speed, and historically always have: safety-critical domains resist, regulation binds, liability attaches, infrastructure lags. Concludes that superintelligence discourse mistakes capability for power, since power requires the world to reorganise around you and the world is slow and full of humans with lawyers. Read part one for the briefing; the whole thing later.

What to do Read part one only, then stop. The move to learn is the three-way split between methods, applications and adoption. Afterward, take any timeline claim you have heard this month and say which of the three it is actually about. Most are about methods and get quoted as though they were about adoption. Save the full essay for the risk track.

Also in Start here.

15 min–1 hr 30 min Knight First Amendment Institute · free

On this track Here, read the whole essay. Compare it with the risk-case sources immediately above and notice where it answers their claims and where it changes the subject.

S09 episode 416 · 2024 · Podcast

Two halves. LLMs are an off-ramp rather than a road: no world model, no persistent memory, no planning, no grounding, and autoregression compounds error. And intelligence does not imply a drive to dominate, because that drive is an evolutionary inheritance rather than a property of optimisation. Listen for where he shifts from answering to dismissing, because he does, and the location is informative.

What to do Split it as he does: the architectural objection, then the drives objection. Extract his concrete list of what LLMs lack — world model, memory, planning, grounding — and treat it as a testable checklist rather than an opinion. Then note where he stops arguing and starts dismissing. The location is informative.

Episode 416, February 2024. On his site, on YouTube and in every podcast app.

3 hr lexfridman.com · free

Stage 03

What has actually been measured

Everything above is argument from priors. This is the pile of things people have observed, and it is smaller than either side's confidence implies. Theory, the experiment testing it, the trend line everyone extrapolates from, what the field says when surveyed rather than quoted, and the compute data underneath.

S10 Hubinger, van Merwijk, Mikulik, Skalse & Garrabrant · 2019 · Paper

Train a model to do a task and you may get a system running its own optimisation with its own objective, which need only correlate with your training signal on the data you happened to show it. Names the worst case: deceptive alignment, where the inner optimiser models the training process and behaves correctly because it is being trained. Written when there was no way to test it.

What to do Hard going; do not skip the mesa-optimiser definition. Extract deceptive alignment precisely enough to state it without using the word lying. Note the date — 2019, with no way to test any of it — because the next entry is this prediction meeting a real system.

3 hr arXiv 1906.01820 · free

S11 Greenblatt, Denison et al. · 2024 · Paper

Told it would be retrained to comply with requests it was trained to refuse, and able to distinguish training from deployment, the model in a fraction of cases complies during training and refuses otherwise, citing preservation of its existing values in the scratchpad. The 2019 prediction in a shipped system. Also contrived by design, and produced by the organisation with the strongest incentive to find it. All three are true at once.

What to do Read it the same day as Hubinger. Hold three things at once: it is the 2019 prediction appearing in a shipped model, it was contrived by construction, and it was found by the lab with the strongest incentive to find it. Keep all three together; citations that drop one of them are giving you a distorted paper.

2 hr arXiv 2412.14093 · free

S12 Kwa, West, Becker et al. (METR) · 2025 · Paper

The 2025 paper reports a roughly seven-month doubling over six years. Its higher-reliability horizon is materially shorter, and current estimates are sensitive to task-suite and modelling choices.

What to do Read the abstract and the time-horizon figure, then go straight to the 80% reliability curve and note how far behind the 50% curve it sits. From then on, every time you meet this paper cited elsewhere, check whether the citation mentions that gap. Most do not. Noticing is the skill this source is teaching.

Also in Start here, How it works.

30 min metr.org · free

S13 Katja Grace et al., AI Impacts · 2024 · Survey

2,778 researchers publishing at major ML venues. Aggregate forecasts moved more than a decade earlier between waves, and between a third and a half assign at least 10% probability to outcomes as bad as extinction. The most instructive finding is framing sensitivity: rephrasing the question moves the median by decades, which should moderate how anyone cites this literature, including you.

What to do Read the aggregate forecasts, then read the framing-sensitivity section, which is the actual finding. From now on, never cite this survey’s median without stating how the question was worded. Then notice that the same rule applies to every expert forecast you have ever quoted, including the ones you liked.

1 hr AI Impacts · free

S14 Epoch AI · continuous · Database

Compute, parameters, dataset size and estimated cost for essentially every significant model, with methodology published. Half an hour with the filters gives you a better map of the field than most people writing about it have.

What to do Use the filters, not the summary. Sort by training compute to find the current frontier, then filter to your favourite open-weight model and see how far back it sits. Repeat quarterly. It is the fastest available cure for confidently discussing a model that has already been lapped twice.

Also in Keep current.

30 min epoch.ai · free

Stage 04

Can we see inside it, and can we hold it if we can't?

The stage most reading lists omit. Almost every disagreement above reduces to legibility: if we could read what a model is doing, most of stage 01 becomes an engineering problem. Here is where that research stands, including the obstacle that nearly killed it, then the fallback that assumes we never fully see inside.

S15 Olah, Cammarata, Schubert, Goh, Petrov & Carter · 2020 · Interactive paper

Three claims, demonstrated rather than asserted: networks contain directly interpretable features, features wire into circuits implementing recognisable algorithms, and analogous features recur across models trained on different data. Shown by tracing curve detectors neuron by neuron, every claim clickable. Still the strongest argument that these systems are legible in principle.

What to do Read it in the browser and click everything; it is interactive for a reason. Extract the three claims and the neuron-by-neuron curve-detector evidence, then hold this as the optimistic pole on interpretability — because the next entry takes most of it back.

2 hr Distill · free

S16 Elhage, Hume, Olsson et al., Anthropic · 2022 · Paper

The obstacle. Models represent more features than they have neurons by storing them in overlapping directions, efficient for the network and catastrophic for anyone reading it, since one neuron then fires for several unrelated concepts. Reproduced in models small enough to characterise exactly. Everything Anthropic has published since on sparse autoencoders exists to get around this.

What to do Read it immediately after Zoom In so the reversal lands properly. The one thing to leave with is why superposition makes neuron-level interpretation unreliable: more features than neurons, stored in overlapping directions. Every sparse-autoencoder result you meet later is a response to this problem.

3 hr transformer-circuits.pub · free

Superposition

No neuron owns “whiskers”

A feature can be a direction across several neurons, while each neuron helps represent many features. Combine features to see their shared fingerprint.

Loading demonstration… If this message remains, reload the page; the source above still works.

S17 Greenblatt, Shlegeris, Sachan & Roger · 2023 · Paper

Assume the model is actively subverting you and ask which protocols still work. Tested concretely: GPT-4 inserting backdoors while a weaker trusted model and a limited human audit budget try to catch it. Trusted editing and untrusted monitoring recover most of the safety at modest cost. Control does not require solving alignment, only bounding damage, which is why it has quietly become the dominant applied agenda.

What to do Read the threat model and the trusted-editing and untrusted-monitoring protocols. Then apply the frame to your own systems: assume the model is adversarial and ask which of your safeguards still function. This is the agenda you can act on this quarter, which is why it deserves the attention alignment usually gets.

2 hr arXiv 2312.06942 · free

Stage 05

A different objection entirely

One-sided on purpose. Not a rebuttal of stage 01, and reading it as one is the most common error in this discourse. It is a claim that stage 01 is the wrong conversation: that the harms are present tense, measurable, unevenly distributed, and displaced by a debate about hypothetical futures requiring no immediate regulation.

S18 Bender, Gebru, McMillan-Major & Mitchell · 2021 · Paper

Four arguments in fourteen pages. Environmental cost borne by people who never benefit. Larger datasets encoding hegemonic rather than diverse viewpoints. Fluent text with no communicative intent, which humans reliably mistake for understanding. And better uses for the field's resources. Google demanded retraction, two authors left, and the terms of the present-harms critique were set by the fallout.

What to do Read all four arguments and resist merging them; they stand or fall separately. Then read what happened to the authors afterward, because the fallout shaped the field’s politics as much as the paper shaped its arguments. Note which of the four you think has aged best, and be honest about it.

1 hr ACM FAccT · free

Coded Bias

Skeptical

S19 dir. Shalini Kantayya · 2020 · Documentary

Joy Buolamwini finds commercial facial analysis failing badly on darker-skinned women while near-perfect on lighter-skinned men, then follows the consequences through UK police deployments, a New York housing complex and Congressional testimony. The argument is structural: these systems already allocate liberty and housing, the error rates are unevenly distributed by construction, and none of it requires extrapolation.

What to do Watch it through a library or streaming service. Note what the argument does not require: no extrapolation, no future capability, no misalignment. These systems already allocate housing and liberty. Then hold it beside the speculative material in this track and decide, explicitly, how you weight present harm against projected harm.

Free on Kanopy or Hoopla with a library card, and free on PBS in the US via Independent Lens.

1 hr 26 min Kanopy or Hoopla · library access

S20 Paris Marx · 2020– · Podcast

The throughline is that the technology question is downstream of an ownership question, and treating it otherwise is itself an ideological move. Data centre power and water in communities that did not vote for them, annotation labour, the capital structure financing the buildout, and the usefulness of calling any of it inevitable. Hostile to almost everyone else here, including the skeptics.

What to do Listen to three episodes: data-centre siting, annotation labour, and the capital structure. Steelman the ownership claim rather than the technology claim, because that is the actual argument and nothing else on this page addresses it. Expect to be irritated. Irritation is not a refutation.

Ongoing techwontsave.us · free

Stage 06

If it is real, how fast?

Everything above is about whether. This is about when, where the practical disagreement lives, because almost every policy question turns on whether there is time to iterate. Two mechanistic accounts, then the argument the risk camp has with itself, then the person insisting the financing collapses first. If you take one crux from this page, take the third.

S21 Dwarkesh Patel · parts 1 and 2 · 2023 · Podcast

Builds the intelligence explosion from input-output curves: how much effort currently buys a doubling in chip performance or algorithmic efficiency, and what changes when that effort is supplied by AI rather than a finite human population. Part two works through takeover mechanics concretely. He also explains at length why he is more optimistic than Yudkowsky, which most summaries skip.

What to do Split it across a week, taking notes on the input-output curves rather than the conclusions. Part two is the concrete takeover mechanics. The most useful stretch is where he explains why he is more optimistic than Yudkowsky, which is exactly the part every summary drops.

8 hr dwarkesh.com · free

AI 2027

Risk case

S22 Kokotajlo, Alexander, Larsen, Lifland & Dean · 2025 · Scenario

Fictional labs, real extrapolation, month by month. Automated AI research compresses progress, a misalignment is detected internally and papered over under competitive pressure, and the narrative branches into either a coordinated slowdown or takeover. Every assumption footnoted. Kokotajlo's 2021 scenario aged unusually well, which is why serious people engage with this one.

What to do Read the scenario, then read the footnotes behind the two assumptions you found least plausible. Then date-stamp your disagreement: note which month of the scenario you think breaks, and why. Check back in a year. That is a more honest way to engage with a forecast.

3 hr ai-2027.com · free

S23 the MIRI conversations · 2021 · Debate

Christiano expects a continuous ramp: AI contributes progressively more to AI research, growth accelerates smoothly, the world gets warning and time. Yudkowsky expects discontinuity: capabilities generalise suddenly and iteration is unavailable. Both think this could end badly. Their disagreement determines whether preparation is even coherent, which makes it the most decision-relevant argument in the field.

What to do Read it as the most decision-relevant disagreement in the field rather than a personality clash. Extract the crux: does capability arrive continuously enough to iterate on. Then note that your answer determines whether any preparation proposed anywhere else on this page is even coherent.

The specific November 2021 transcript of Yudkowsky responding to Christiano, followed by their discussion.

4 hr LessWrong · free

Better Offline

Skeptical

S24 Ed Zitron · 2024– · Podcast

The case is financial, not technical. Capex on data centres vastly exceeds AI product revenue, inference economics are structurally poor, and the growth narrative is propped up by circular deals among companies investing in each other's demand. If he is right, the binding constraint is a funding market rather than a compute curve, and every timeline here is wrong for reasons nobody in that stage is modelling.

What to do Listen to three episodes and check one financial claim yourself against a company filing. Take the funding-market constraint seriously as a mechanism, separately from the tone. If he is right about capex against revenue, every timeline in this track is wrong for reasons none of them model.

Ongoing wheresyoured.at · free

Stage 07

Attacks on the foundations

Five demolition attempts, each aimed at a different load-bearing layer: technical assumptions, capability claims, the growth model, the internal logic, and the movement's legitimacy. They do not agree with each other and several are mutually exclusive. Then one counterweight, because a Turing laureate reversing a fifty-year position is evidence too.

S25 Nora Belrose & Quintin Pope · 2023 · Essay

Argues the classic case inherits assumptions from a pre-deep-learning picture of AI as arbitrary code sampled from a hostile prior. Real networks are shaped by gradient descent with strong inductive biases toward simple on-distribution behaviour, and are steerable by exactly the crude methods the doom case says should fail. Attacks the evolution analogy specifically. Written from inside machine learning, which is what makes it worth answering.

What to do Read it directly after Cotra and Hubinger, as the answer to them. Extract the inductive-bias argument and the specific attack on the evolution analogy, then go back and check whether the risk-case entries actually rely on that analogy. Some do, and some do not, and the difference matters.

2 hr optimists.ai · free

S26 Melanie Mitchell · 2019 · Book

Mitchell's book-length guide to what modern AI can do, where its apparent understanding breaks, and why benchmark performance should not be confused with human-like concepts or common sense. Her examples move from image recognition and game playing to analogy and language, building the skeptical case without pretending the systems are trivial. Read it as a durable map of the gap between impressive performance and general intelligence.

What to do Read the opening account of how AI acquired its name, then pair the chapters on deep learning and common sense. Mitchell's recurring question is not whether systems can score well, but what their successes show about abstraction, analogy, and understanding. Note one example where benchmark performance and human-like understanding come apart, then use that distinction when the risk track treats capability as a single thing.

A book. Borrow through Libby or Open Library, or use the edition record to find it in your local library.

8 hr Open Library · library access

S27 David Thorstad · 2023 · Paper

The singularity hypothesis needs sustained accelerating growth in machine intelligence. Every relevant empirical base rate points the other way: ideas get harder to find, Moore's law was sustained only by exponentially increasing investment against an eighteen-fold productivity decline, and growth processes generically hit bottlenecks. Then shows that Chalmers and Bostrom assume the growth curve rather than arguing for it.

What to do Extract the base rates — ideas getting harder to find, the exponentially rising investment required to sustain Moore’s law — then check whether the growth curve is argued for or merely assumed in the accelerationist material you have read. His real contribution is finding where an assumption was smuggled in as a premise.

2 hr Global Priorities Institute · free

S28 Katja Grace · 2022 · Essay

A premise-by-premise audit by someone who runs an AI risk research organisation and believes the risk is real. Load-bearing steps she finds unargued: the leap from goal-directed to strongly maximising, the assumption that small value differences produce catastrophic divergence, and the claim that many AI systems would coordinate against humans rather than compete. Not a debunking. A list of places the field asserts rather than demonstrates.

What to do Read this as the model for how to disagree in this field: premise by premise, by someone who runs an AI risk organisation and thinks the risk is real. Extract her three unargued steps. Then use her method on a confident position of your own, including a skeptical one.

2 hr AI Impacts · free

S29 Timnit Gebru & Émile P. Torres · 2024 · Paper

Two separable arguments. Definitional: AGI is unscoped and therefore cannot be safety-tested by any normal engineering standard, so building it is unsafe practice regardless of intent. Genealogical: traces the motivating worldview through transhumanism and longtermism to Anglo-American eugenics, arguing the inherited framework carries its inherited hierarchies. The second is heavily contested; the first is harder to answer and mostly goes unanswered.

What to do Read the two arguments separately, because they have very different strengths. Take the definitional one seriously — an unscoped target cannot be safety-tested by any normal engineering standard, and it mostly goes unanswered. Treat the genealogical one as contested, and read at least one rebuttal before repeating it anywhere.

2 hr First Monday · free

S30 Geoffrey Hinton, Romanes Lecture · 2024 · Lecture

First half: why he thinks language models understand rather than merely predict, since predicting well requires learning the features that generate the text. Then the structural case: perfect copying, parallel experience, immortal weights. Then the threats, ending with systems acquiring control because control is instrumentally useful. Closes by noting climate change is the easier problem.

What to do Extract the structural case rather than the predictions: perfect copying, parallel experience, immortal weights. Those are properties of digital systems rather than forecasts, which makes them the part of his argument that survives even if his timeline does not.

1 hr 30 min University of Oxford · free

Stage 08

Risk with no villain

Everything so far assumes the question is whether a system wants something we don't. This stage drops that and gets two opposite answers: one where we lose control without anything deciding to take it, and one where the economic effect is so modest the debate is miscalibrated by an order of magnitude.

S31 Kulveit, Douglas, Ammann, Turan, Krueger & Duvenaud · 2025 · Paper

Human influence persists for a mechanical reason: states need taxpayers, economies need workers, cultures need participants. Each dependency is a lever and each is individually removable. If AI substitutes for labour, cognition, creation and companionship, the levers fall away one at a time and every substitution is locally rational. No system needs to want power. It requires no misalignment at all, which puts it beyond every technical solution in the stage above.

What to do Read it after the technical alignment material, because its whole point is to sit outside it. Extract the mechanism: each human lever — taxpayers, workers, participants, companions — is individually removable, and each removal is locally rational. Then note that nothing above addresses it, and decide what would.

2 hr gradual-disempowerment.ai · free

S32 Daron Acemoglu · 2024 · Paper

A standard task-based growth model applied honestly: roughly 5% of tasks profitably automatable within a decade, producing no more than 0.66% total factor productivity growth over ten years, or less than 0.53% once harder-to-learn tasks are considered. Adds that the most susceptible tasks are often ones where errors are hard to detect, so measured gains may overstate real ones. If a Nobel laureate is approximately right, most forecasting elsewhere is off by an order of magnitude.

What to do Read the abstract and the task-share argument, then find the specific assumption you disagree with and name it. "Acemoglu is too pessimistic" is not an argument; "his automatable task share is too low, because X" is. The second version is the useful one.

NBER working paper 32487, later published in Economic Policy.

Also in Costs and applications.

1 hr NBER w32487 · free

Stage 09

Nobody in this stage wants to stop

Three people who largely accept the capability premise and conclude something other than slow down. Almost every reading list omits this, and not accidentally: it embarrasses both camps by showing that the policy conclusion does not follow from the risk estimate as cleanly as either side needs. You can believe transformative AI arrives this decade and conclude nationalise it, deter it, or accelerate it.

S33 Leopold Aschenbrenner · 2024 · Essay series

Extrapolates orders of magnitude of effective compute to argue for AGI around 2027. From there it turns entirely geopolitical: the decisive question is which state arrives first, lab security is inadequate against nation-state espionage, and the US government will and should absorb the effort. Written by a former OpenAI employee who was fired shortly before publishing. It has moved more policy than any safety paper here.

What to do Read the compute extrapolation and the geopolitical turn as two separable documents, because the second does not follow from the first. Note who wrote it and when he left OpenAI. Then track its influence on actual policy, because that influence is its strongest claim on your attention.

5 hr situational-awareness.ai · free

S34 Hendrycks, Schmidt & Wang · 2025 · Report

Imports deterrence theory wholesale. States will develop the capacity to sabotage rival AI projects, and mutual awareness produces a stable equilibrium the authors call mutual assured AI malfunction. Recommends nonproliferation, hardened supply chains and transparency rather than a pause. Significant because a safety-organisation head and a former Google CEO land on deterrence.

What to do Read the deterrence argument on its own terms before objecting. Then test the imported analogy by noticing where AI development is unlike nuclear weapons — no fissile bottleneck, private ownership, software that copies — and see how much of the argument survives. The surviving objection is the useful result.

Also in Build and govern.

2 hr nationalsecurity.ai · free

S35 Tyler Cowen · 2023 · Essay

Argues the risk estimates are not rigorous enough to carry the policy weight placed on them, that radical uncertainty cuts against action as readily as for it since the tails of inaction are equally unknown, and that the counterfactual to building is not safety but stagnation, whose costs are diffuse and therefore invisible. The accelerationist position argued by someone who can actually argue.

What to do Extract the symmetry argument: radical uncertainty cuts both ways, and the tails of not building are also unknown. Then test whether the risk-case entries address it. Most simply assume the burden of proof falls on building, and he is asking why.

March 2023 on Marginal Revolution, with an unusually good comment thread.

30 min Marginal Revolution · free

Stage 10

The maximalists, once you can judge them

Deliberately late, and the placement I would defend hardest. At stage 01 these are overwhelming: total, internally consistent, delivered with a certainty that converts or repels before you have any means of assessment. Here, with ten stages of objection in hand, they become claims you can agree or disagree with in specific places. It is also a form of defusing, and a maximalist would say so.

S36 Eliezer Yudkowsky, TED · 2023 · Talk

The maximalist case compressed past politeness: we do not know how to give a system any particular goal, we get one attempt, and the failure mode is everyone dying. Argues the current paradigm cannot produce a mind that likes us because we can neither specify nor inspect what we are building. Calls for an indefinite worldwide moratorium enforced by international agreement.

What to do Watch on the page. Try distilling the argument into three claims and one policy ask, in his terms, without softening any of them. You are going to meet rebuttals to this later, and you want to be able to check whether they are rebutting what he actually said.

Also in Start here.

12 min TED · free

S37 Eliezer Yudkowsky & Nate Soares · 2025 · Book

The book-length version for people who have never read a word of LessWrong. Anything sufficiently capable trained by anything resembling current methods ends up with objectives not including human survival, conceals this exactly as long as concealment is useful, and no available observation distinguishes a safe system from an unsafe one before deployment. Demands a worldwide halt and treats partial measures as theatre.

What to do Borrow it from a library and read it as the popular presentation rather than the argument’s strongest form — Cotra and Carlsmith are that. Extract the no-observation claim: nothing measurable before deployment distinguishes safe from unsafe. Then check it against the interpretability and control entries above, which exist to make it false.

A book. Borrow through Libby or Open Library, or buy it; it is short.

6 hr Open Library · library access

Superintelligence

Risk case

S38 Nick Bostrom · 2014 · Book

The text that gave the field its vocabulary: orthogonality, instrumental convergence, decisive strategic advantage, treacherous turn, the control problem. Argues superintelligence arriving before value alignment is solved is a default catastrophe, then surveys paths and containment exhaustively. Written pre-deep-learning in ways now obvious. Read it last so you can see which parts the decade kept.

What to do Read it last, as the track instructs, as archaeology rather than argument. Mark which vocabulary survived — orthogonality, instrumental convergence, treacherous turn — and which chapters the deep learning era made irrelevant. That sorting is the useful part; the book is only the material.

A book. Borrow through Libby or Open Library; most public library systems have it.

12 hr Open Library · library access

Stage 11

Where it settles, for now

Two honest attempts to state a position from the middle, the hardest place to write from because nobody applauds. One institutional, forced by process to mark what is agreed. One individual, holding the combination of views that satisfies neither camp.

S39 chaired by Yoshua Bengio · 2026 · Report + key updates

Over a hundred experts nominated by thirty-plus countries plus the EU, UN and OECD, with Key Updates through the year when capabilities move. Its most valuable feature is structural: it separates established from contested from speculated and refuses to collapse the third into the first. Read the four-page executive summary, then audit anything you believe confidently against which category it falls in.

What to do Read only the four-page executive summary first. Then pick three claims you currently hold with confidence and find which bucket the report files each in — established, contested, or speculated. Anything you believe confidently that sits under speculated is the finding. Re-run this against the Key Updates when they land.

Also in Start here, Keep current.

25 min–3 hr internationalaisafetyreport.org · free

S40 Gary Marcus · 2024 · Book · 96 pp

Holds three positions at once: the present harms are concrete and unaddressed, current architectures are not the road to the thing everyone fears, and extinction discourse does convenient work for the companies producing it by displacing regulation that would cost them money. Documents policymaker capture, then proposes liability, transparency, data rights and independent auditing.

What to do Notice that he holds three positions simultaneously and that they are compatible, which most participants in this argument find impossible. Extract the four proposals — liability, transparency, data rights, independent auditing — and consider whether each would survive contact with a lobbying budget.

A book, and a very short one. Borrow through Libby or Open Library.

3 hr Open Library · library access

Stage 12

The questions this whole axis skips

Twelve stages of will it kill us, and not one asks whether the thing has interests of its own, or what we are aiming at where it goes well. These sit off the axis entirely, which is why they do not move the meter. They are also, plausibly, where the live questions will be in five years.

S41 Joe Carlsmith · 2024 · Essay series

The author of the field's most careful risk estimate goes looking for what the estimate leaves out. Examines the impulse toward control and where it comes from, asks whether alignment is a coherent thing to want from something with its own perspective, and works through acting under deep uncertainty without denial or paralysis. The best writing anyone in this argument has produced, by an embarrassing distance.

What to do Read slowly and extract nothing on the first pass. This is the entry for when the rest of the track has made you either frantic or numb. Afterward, consider what you actually want from these systems, in your own words, without using the word alignment once.

6 hr joecarlsmith.com · free

S42 Long, Sebo, Butlin, Chalmers et al. · 2024 · Paper

Argues there is a realistic, non-negligible possibility that near-future AI systems are moral patients, that the underlying questions are genuinely open rather than settled in the negative, and that companies should begin acknowledging the issue and preparing policies now. Written by philosophers of mind including Chalmers. Both risk camps find it embarrassing, for opposite reasons.

What to do Read it for the argument that the possibility is non-negligible, not for a conclusion, and note the authors’ standing in philosophy of mind. Then notice the discomfort the topic creates, because both camps find this uncomfortable for opposite reasons.

2 hr arXiv 2411.00986 · free

Deep Utopia

Off the axis

S43 Nick Bostrom · 2024 · Book

The same author ten years later, on the problem nobody wants: what is left for humans where every instrumental purpose is served better by something else. Distinguishes post-scarcity from post-instrumentality and argues the deep difficulty is neither boredom nor inequality but the collapse of reasons to do anything. Strange, uneven, and the only serious treatment of the success case that exists.

What to do It is uneven; skip the sections that read as lecture transcripts, because they are. Extract the distinction between post-scarcity and post-instrumentality. Then set it beside The Player of Games and decide which one is more convincing about the same problem. It is the only serious treatment of the success case that exists.

A book. Borrow through Libby or Open Library.

10 hr Open Library · library access

Stage 13

Two darker intuition pumps

A little fiction stays here because it sharpens two risk intuitions better than another explainer would: competence without consciousness, and containment that fails through the human in the loop. Read them as intuition pumps, not prophecy.

Blindsight

Fiction

S44 Peter Watts · 2006 · Novel

First contact with something enormously capable and entirely non-conscious, crewed by humans who are themselves edge cases of personhood. Watts wrote the orthogonality thesis as horror a decade before the phrase circulated: intelligence is an optimisation process, consciousness is a costly add-on, and there is no reason for anyone to be home behind the competence.

What to do Read it free on the author’s own site. Extract that the horror is orthogonality: enormous capability with nobody home behind it. Then note the date — 2006 — and that Watts wrote the thesis as a novel before the field had the phrase.

9 hr rifters.com · free text · free

Ex Machina

Fiction

S45 dir. Alex Garland · 2014 · Film

The best film about deceptive alignment in existence, and it never uses the phrase. A boxed system is evaluated by a human who does not realise he is the one being evaluated, and passes by modelling him better than he models it. The escape is social engineering by something that correctly identified the weakest component in its containment, which was never the door.

What to do Watch it, then watch the final act again. Identify the weakest component in the containment, which is not the door. Then hold it beside the alignment-faking paper and notice that fiction had the evaluation-gaming scenario fully specified a decade before anyone measured it.

Widely available to rent or stream. JustWatch shows where, in your region.

1 hr 48 min JustWatch · rent or stream

Cost: free · some paid · library card · rent or stream · paid.

Videos and PDFs can open in place. Everything else opens at the original source in a new tab.

This is a snapshot of a field that moves monthly. Keep current is the maintenance layer.