Calibrated Estimation

Definition

A person is "calibrated" if, across many estimates, their stated confidence matches their actual hit rate: of all the times they give a 90% confidence interval (CI), the true value should fall inside it about 90% of the time. Calibration is a measurable, trainable skill, not a personality trait — and the overwhelming default, absent training, is overconfidence: stated ranges are too narrow and stated probabilities too extreme relative to how often the person is actually right.

In the Book

Chapter 5 walks through Hubbard's calibration test: ten trivia questions requiring a 90% CI (e.g., the year Newton was born) plus ten binary questions with a stated confidence level. Drawing on decision-psychology research by Daniel Kahneman and Amos Tversky, Hubbard reports results from 927 individuals who took his firm's calibration training between 1996 and the book's writing: a perfectly calibrated person should get 7–10 of 10 range questions right, yet only 29% of subjects managed this, and 24% got 3 or fewer right — a score so unlikely for a calibrated person (under 1 in 100,000) that it demonstrates severe, near-universal overconfidence. Even worse, over 15% of answers where people stated "100% confidence" were wrong. Hubbard then describes the fix: training methods including the "equivalent bet" test (would you rather bet on your stated range being right, or spin a wheel with the same odds?) and repeated practice with feedback, which reliably move people toward true calibration — a result he contrasts with odds-makers and bookies, who are professionally forced to calibrate and do measurably better than executives or physicians making similar probabilistic judgments (e.g., the odds a tumor is malignant).

Why It Matters

Calibrated ranges are the raw material every later step in the book depends on: you can't compute the value of a measurement, run a Monte Carlo simulation, or decide what's worth measuring further without an honest starting estimate of what you already know. More broadly, calibration training demonstrates that "how confident should I be" is not introspective guesswork but an empirically checkable, improvable skill — which reframes expert judgment itself as something that can be measured, audited, and trained rather than simply trusted or dismissed.