Chapter 14 — The Strategic Picture¶

Core Thesis¶

Zooming out from the mechanics of control and value-loading to the broader question: what should we actually do, given all this? Bostrom introduces general tools for long-term science and technology strategy — evaluated mostly from the impersonal perspective (weighing all lives equally regardless of when they occur, valuing new happy lives created) rather than the person-affecting perspective (weighing only currently-or-independently-existing people) — and applies them to the specific strategic terrain of an intelligence explosion.

The Mechanism — Differential Development and Preferred Order¶

Against the "futility objection" (research can't be stopped, so opposing any given line is pointless — an argument researchers invoke to defend funding but never to argue for cutting their own), Bostrom's principle of differential technological development: retard dangerous technologies, accelerate beneficial ones, regardless of whether the "technological completion conjecture" (everything obtainable eventually gets obtained) is true — because when, by whom, and in what order technologies arrive still matters enormously even if their eventual arrival doesn't depend on us. This yields a preferred order of arrival argument: superintelligence would eliminate most natural existential risks (asteroids, supervolcanoes) and many anthropogenic ones (accidental catastrophe, coordination failures, war), so — all else equal — getting superintelligence before other dangerous technologies like advanced nanotechnology is better than the reverse, since a superintelligence-first world only ever faces superintelligence's own risk, while a nanotech-first world faces nanotech's risk and then superintelligence's risk on top.

State Risk vs. Step Risk¶

Bostrom's crucial risk-accounting distinction: a state risk (asteroid strikes, nuclear war under prolonged anarchy) accumulates the longer a system remains exposed to it — running faster through a dangerous era helps. A step risk (the intelligence explosion itself, conditional on fast takeoff) is a one-time transition risk roughly independent of how quickly it's traversed — "one does not halve the risk of traversing a minefield by running twice as fast." Since current existential state risk looks low on a decade timescale, but the intelligence explosion is overwhelmingly a step risk, the practical upshot is that speed mostly matters through preparedness at the critical juncture — more time before the transition mostly helps because it means more progress on the control problem, not because delay itself reduces risk.

Cognitive Enhancement and Technology Coupling¶

Should we accelerate human cognitive enhancement to help solve the control problem faster? Bostrom's answer turns on what kind of problem the control problem is: if progress depended mainly on accumulated experience (like learning to avoid war through trial and error), a slower pace with more elapsed calendar time would help more than raw intelligence. But the control problem is a foresight-and-theory problem with essentially no possibility of direct prior experience — so what matters is total intellectual progress achieved by the time of detonation, and cognitive enhancement plausibly increases that total (partly because control-problem progress may be even more bottlenecked on raw intellectual horsepower than AI-building progress itself, and partly because only sufficiently capable minds reliably recognize the problem is worth prioritizing at all). Technology coupling adds a wrinkle: pushing whole brain emulation research toward completion could instead produce neuromorphic AI as an unplanned byproduct — a design pattern that borrows biological structure without the deep mathematical understanding that would make it safe, plausibly more dangerous than either full emulation or clean synthetic AI. "Before you marry your sweetheart, consider the prospective in-laws."

Collaboration and the Race Dynamic¶

A race dynamic — teams fearing being overtaken, even if no real rival exists (the Allies built the bomb faster believing, wrongly, that Germany was close) — systematically shifts investment away from safety toward speed. Box 13's models formalize an ugly result: race dynamics get worse, not better, the more competitors know about their relative standing — a leader who knows it's ahead can afford more safety investment, but a laggard who knows it's behind is pushed to cut corners to catch up, and in expectation the second effect dominates the first. More competitors also uniformly increase danger. The remedy Bostrom proposes is the common good principle — "superintelligence should be developed only for the benefit of all of humanity and in the service of widely shared ethical ideals" — operationalized concretely via a windfall clause: any firm or state whose profits or GDP share ever crosses an extreme, near-impossible threshold commits in advance to distributing the excess to everyone. The proposal is nearly costless today (the threshold will almost certainly never bind) while giving humanity real insurance against any single actor "hitting the jackpot" with the intelligence explosion and keeping the entire cosmic endowment to themselves.

Key Terms¶

Impersonal vs. person-affecting perspective — weighing all possible lives equally vs. weighing only those who exist or will exist regardless of the choice
Differential technological development — deliberately slowing dangerous and hastening beneficial lines of research
Preferred order of arrival — getting risk-reducing technologies (superintelligence) before other dangerous ones (nanotech)
State risk vs. step risk — accumulating exposure risk vs. one-time transition risk largely independent of speed
Technology coupling — pursuing one technology reliably producing another as a side effect, intended or not
Race dynamic — competitive pressure that trades safety investment for speed
Common good principle / windfall clause — a pre-committed norm and mechanism for universally sharing the gains of superintelligence