Toby Ord

“Firing the nuclear weapon required the agreement of the submarine's political officer, who held the other half of the firing key. Despite the lack of authorization by Moscow, the political officer gave his consent. On any of the other three submarines, this would have sufficed to launch their nuclear weapon. But by the purest luck, submarine B-59 carried the commander of the entire flotilla, Captain Vasily Arkhipov, and so required his additional consent. Arkhipov refused to grant it.”

“And they require foresight rather than trial and error. Since they allow no second chances, we need to build institutions to ensure that across our entire future we never once fall victim to such a catastrophe.”

“it fails completely, for by their very nature, we never have any experience of existential catastrophe before it's too late.”

“And the total natural risk, around 1 in 10,000. Nuclear war, around 1 in 1,000. Climate change, around 1 in 1,000. Other environmental damage, around 1 in 1,000. Naturally arising pandemics, around 1 in 10,000. Engineered pandemics, around 1 in 30. Unaligned artificial intelligence, around 1 in 10. Unforeseen anthropogenic risks, around 1 in 30. Other anthropogenic risks, around 1 in 50. And the total anthropogenic risk, around 1 in 6. And the total existential risk, around 1 in 6.”

“risks, such as disease, war, tyranny, and permanently locking in bad values, are correlated across different planets. If they affect one, they are somewhat more likely to affect the others too. A few risks, such as unaligned AGI and vacuum collapse, are almost completely correlated.”

“the relative loss each year is actually rather slow, about one part in five billion, and it is this relative reduction in our potential that matters. The pressures towards prudence and wisdom are larger in relative terms, and counsel care over haste. If rushing to acquire these technologies a century earlier diminished our chance of survival by even one part in fifty million, it would be counterproductive”