There's a particular kind of confidence that comes from writing a number next to a fear. "5% chance of human extinction from AI." "One in a thousand." "10-20% by 2100." These figures circulate through think pieces, conference talks, and policy briefs as if they were measured rather than imagined. They feel rigorous. They invite comparison, budgeting, prioritization.
They are, in an important sense, a category error.
The Temptation of the Number
We like probabilities because they promise to tame uncertainty. Roll a die enough times and the frequencies stabilize. Count the failures in a batch of widgets and you can bound the defect rate. This is the machinery of insurance, medicine, and engineering — and it works because it rests on a foundation of repeated, comparable events.
Now ask: how many human civilizations have been destroyed by artificial intelligence?
One is not a sample. Zero is not a sample. It is a single case, still unfolding, with no siblings to compare against. When we say "5% chance," we are not reporting a frequency. We are reporting an intuition — sometimes a very well-informed one, but an intuition nonetheless — dressed in the language of statistics.
Four Reasons the Math Breaks
1. The reference class problem.
To estimate a probability, you need a population of similar cases. What is the population here? "Advanced civilizations"? We have one data point, and it's us. "Powerful new technologies"? The set is tiny, wildly heterogeneous, and mostly didn't have the property of being able to improve themselves. There is no honest denominator.
2. The one-shot problem.
Even if we could estimate an annual risk, humanity faces an absorbing state. If the bad outcome occurs, there are no further trials. You cannot average over outcomes when one outcome removes you from the game. A "small" annual risk compounded across centuries doesn't stay small — and there is no recovery, no learning curve, no second attempt.
This is what Nassim Taleb calls the ruin problem: expected value reasoning collapses when the losing branch is terminal. A strategy that wins 99 times and dies once is not a 99% strategy. It is a dead strategy.
3. The uncertainty isn't in the model — it's in the model choice.
When a forecaster says "5%," the honest error bars aren't 4–6%. They're closer to "somewhere between negligible and near-certain, depending on assumptions we can't verify." The uncertainty about which model is right dwarfs the uncertainty within any given model. Reporting the latter while hiding the former is not precision. It's theater.
4. Reflexivity.
Dice don't read the odds. Institutions do. Publish "5% chance of doom" and you change funding priorities, regulation, and the speed of development. The prediction becomes a cause. Standard probability theory has no clean place to put this — the system being modeled is watching the model.
What the Number Is Actually Doing
None of this means the concern is irrational. It means the number is doing rhetorical work, not scientific work.
When someone writes "10%," they usually mean something like: this is serious, non-negligible, and deserves attention comparable to other existential threats. That's a defensible claim. But it doesn't require a probability. And dressing it as one invites three failure modes:
· False precision — the number gets quoted, then re-quoted, until it hardens into fact.
· False comparisons — "AI risk is 5%, climate is 3%, so fund AI" treats incomparable things as commensurable.
· False confidence — it implies we know the outcome space. We don't. Genuine unknowns can't be assigned meaningful probabilities in advance.
What to Do Instead
If probabilities are the wrong tool, what's the right one?
Think in directions, not magnitudes. Is the risk increasing or decreasing? Are capabilities outpacing safeguards? Is the window for intervention widening or closing? These questions are answerable without inventing a number.
Optimize for robustness, not expected value. Under deep uncertainty, the goal isn't to maximize the average outcome — it's to avoid the catastrophic tail. Prefer reversible moves. Build redundancy. Keep options open.
Use decision rules for ambiguity. Maximin (avoid the worst case), minimax regret (minimize how much you'd wish you'd acted differently), and the precautionary principle all work without a probability estimate. They are designed for exactly this situation: catastrophic, irreversible, uncertain.
Stress-test scenarios. Instead of asking "how likely is this?", ask "what would have to be true for this to happen, and what would we do?" Scenario planning doesn't require knowing the odds. It requires imagination and preparation.
Hold the humility. The honest position is not "it's 5%" or "it's 0.001%." It's: we don't know, the stakes are asymmetric, and the cost of being wrong in one direction is permanent.
The Point Isn't the Number
The debate over AI risk keeps getting dragged into fights about decimal places. That's a distraction. The real questions are structural: How much of our future are we willing to concentrate in systems we don't fully understand? What safeguards are worth building even if the risk is small? Who decides?
Probabilities are a tool for managing repeated, bounded, recoverable risks. Extinction is none of those things. Applying the tool anyway doesn't make us rigorous. It makes us feel rigorous while we avoid the harder work of deciding what to do.
If you take one thing from this: when someone hands you a probability for the end of the world, ask what model produced it, what reference class it assumes, and whether the person quoting it would accept the same reasoning for a risk they don't already believe in.