A Demonstration Is Not a Test

Definition

A test is designed to reveal a system's actual behavior, whatever that turns out to be; a demonstration is designed in advance to make a system look good and to avoid surprises. The two produce superficially similar artifacts — someone runs the software and reports results — but they answer opposite questions, which is why demonstrations can be dressed up as tests to manipulate a buyer's confidence.

In the Book

Weinberg recalls designing performance-benchmark "tests" earlier in his career and admitting that for any two competing systems he could always construct a benchmark that "proved" his employer's product superior — and could have proved the opposite working for the competitor. He dramatizes the pattern with a sales scenario: vendor DataSpew offers buyer Laverne a week-long side-by-side trial with its own "performance engineer," Phyllis, who writes and runs ten test reports and finds DataSpew faster in all ten — a rigged outcome dressed as an independent test, which Weinberg names The Performance Benchmark Scam, a species of The Demonstration Scam. He extends the same logic to vendor testimonials and certifications, comparing them to paid celebrity endorsements on bottled water, and to his own earlier observation that stacking three tools each claiming 50-80 percent cost savings would mathematically shrink your testing budget to near zero — a tell that the claims can't all be independently true. His rule of thumb: good tools amplify whatever effectiveness you already have, for better or worse, and any vendor claim beyond that is likely a scam.

Why It Matters

This is a portable detector for a con that recurs anywhere someone with a stake in the outcome controls the conditions of the "test" — sales pitches, benchmark marketing, curated case studies, even self-reported audits. The tell is the same everywhere: was this evaluation designed to find out what's true, or designed by an interested party to avoid finding out something inconvenient? If the entity being evaluated also designed and ran the evaluation, treat the result as a demonstration, not evidence.