existential risk breaks trial and error because the first failure is the last event
Every adaptive system -- evolution, markets, science, startups -- works by trying things, observing outcomes, and adjusting. The hidden assumption: failures are survivable. Evolution requires organisms to die, not species. Markets require companies to fail, not the economy. Science requires hypotheses to be falsified, not the laboratory destroyed.
Existential risks violate this assumption. A nuclear war, a misaligned superintelligence, a catastrophic pandemic, or irreversible ecological collapse are failures from which the system cannot recover to try again. The first instance of the failure is also the last instance of anything. Trial and error works because errors are informative -- but existential errors cannot inform because there is no one left to learn.
This is not an argument against risk-taking. It is an argument for categorical separation between risks that are survivable (and therefore learnable) and risks that are terminal (and therefore must be prevented a priori). Taleb's "Antifragile" framework makes this precise: systems should be antifragile (gaining from volatility) at the level of components but absolutely robust at the level of the whole. Individual firms should fail; the economy should not. Individual experiments should go wrong; civilization should not.
The implication for governance is that existential risks cannot be managed through normal institutional processes that were designed for recoverable failures. Democratic deliberation is too slow. Market signals come too late. Scientific consensus forms after observation, but there will be no second observation. This creates a fundamental tension: the precautionary principle is both necessary (for existential risks) and paralyzing (if applied to all risks). The resolution requires distinguishing between risks by their recoverability, not their probability.