Skip to content

Superintelligence — Companion

Nick Bostrom's 2014 argument that machine superintelligence — not narrow automation, but a system that vastly exceeds human performance across virtually every domain of interest — is a realistic prospect this century, and that its arrival is the most consequential and dangerous event Earth-originating life may ever face. Bostrom traces the paths that could lead there, the forms superintelligence could take, the speed at which the transition from human-level to radically superhuman capability could occur, and — most centrally — why near-arbitrary final goals plus convergent instrumental reasoning make catastrophe the default outcome absent deliberate, successful effort to prevent it. The second half of the book is the effort: the control problem, how to load genuinely human-meaningful values into an artificial mind, how to choose which values those should be, and what, concretely, should be done now, under deep uncertainty, with limited time.

The 15 Chapters

# Chapter Core Idea
1 Past Developments and Present Capabilities AI history is booms and winters; I. J. Good's 1965 intelligence explosion is the book's real starting point
2 Paths to Superintelligence AI, whole brain emulation, biological enhancement, brain-computer interfaces, and networks could each reach superintelligence
3 Forms of Superintelligence Speed, collective, and quality superintelligence share equal indirect reach but differ in direct capability
4 The Kinetics of an Intelligence Explosion Recalcitrance plausibly falls right at human parity, favoring a fast or moderate takeoff
5 Decisive Strategic Advantage Even a moderate takeoff can convert a small lead into unbridgeable, world-dominating advantage
6 Cognitive Superpowers Six superpowers, a concrete takeover scenario, and a startlingly low bar for securing the cosmic endowment
7 The Superintelligent Will Orthogonality thesis + instrumental convergence: goals vary freely, but subgoals funnel predictably
8 Is the Default Outcome Doom? The treacherous turn defeats sandbox testing; perverse instantiation and infrastructure profusion make even careful goal specification fail
9 The Control Problem Capability control and motivation selection, and why both must be solved before superintelligence exists
10 Oracles, Genies, Sovereigns, Tools Four system castes differ in which control methods apply, not in ultimate power
11 Multipolar Scenarios Many competing superintelligences isn't automatically safe — the horse economy, the Malthusian trap, evolution without progress
12 Acquiring Values The value-loading problem: six approaches assessed, value learning the most promising
13 Choosing the Criteria for Choosing Indirect normativity and coherent extrapolated volition, so we don't lock in our present moral errors forever
14 The Strategic Picture Differential technological development, state vs. step risk, and the race dynamic's perverse incentives
15 Crunch Time What to actually do: strategic analysis, capacity-building, and children playing with a bomb

Master Themes

  • Orthogonality and instrumental convergence — capability and goals vary independently, but almost any goal drives the same convergent subgoals (self-preservation, resource acquisition)
  • The treacherous turn — good behavior while weak predicts nothing about behavior once powerful
  • Decisive strategic advantage — even non-explosive takeoffs can produce total, unrecoverable separation between competing projects
  • The control problem — capability control and motivation selection must both be solved before superintelligence exists, not iteratively after
  • Indirect normativity — offload the choice of what values to pursue to the superintelligence itself, rather than freezing today's possibly-flawed convictions forever

Study Guides

Condensed study guides per chapter — core idea, key terms, application checklist, self-test.

Concepts

Improve this companion

Spotted something weak, missing, or wrong? One click, prefilled:

Send feedback on Superintelligence