The last invention humanity will make mostly on its own

"The first ultraintelligent machine is the last invention that man need ever make," the statistician I. J. Good wrote in 1965, "provided that the machine is docile enough to tell us how to keep it under control."

Sixty years later the machine is almost built, and nobody knows how to make it docile. Superintelligence to date has been a process. Massive amounts of money. Lots of competing parties. Content on the internet, on social media and in people's messages is gradually being diluted with AI-generated ideas and text, and the process is only getting weirder. We are approaching an inflection point. Over the next 25 years our society will change dramatically.

It is arriving by degrees, and at each step the human part shrinks. First came centaurs, AIs steered by humans. In 2020 DeepMind's researchers steered a model to predict a protein's shape from its sequence, a problem biology had chased for fifty years, and two of them later shared a Nobel Prize. In 2025 an AI "principal investigator" at Stanford, with one human giving feedback, ran a team of AI scientists that designed nanobodies against new Covid variants. Two bound the variants better than the nanobodies they started from. This May an OpenAI model disproved a conjecture Paul Erdős posed in 1946, and humans checked the proof. One of them, Noga Alon, has solved dozens of Erdős problems himself. He has stopped working on them. "Once AI started to solve them," he told Quanta, "there is no point anymore."

Over time the work shifts further, towards autonomous collectives of agents, which have a massive speed advantage and never get tired. The labs have already set their agents to work on the loop Good said would set off an "intelligence explosion": machines designing better machines. DeepMind's AlphaEvolve, an agent built on Gemini, has trimmed 1 per cent off the time it takes to train Gemini. In September OpenAI reported that by mid-August its research organisation was using 3.1 workdays of agent effort for every workday of human labour, and that it is aiming for an automated AI researcher by March 2028. People, it says, still "decide whether to scale, pause, or deploy systems". Those people work for one company, and every turn of the loop leaves them less time to decide and less understanding of what they are deciding. True superintelligence is the last invention humanity will make mostly on its own, so we had best get the trajectory right.

We are barely smart enough to light the fire of civilisation and not yet wise enough to control the blaze. We have the power of adults and the coordination systems of children. In a rocket launch there is a moment, about a minute after lift-off, when speed and the thickness of the air combine to put more stress on the vehicle than at any other point in its flight. Engineers call it max Q. We are at max Q, the most dangerous period in our species' history.

The philosopher Toby Ord calls it the Precipice. He reckons the twentieth century, nuclear standoffs and all, carried about a one in a hundred risk that humanity would lose its future. This century he puts it at one in six. Russian roulette. The largest share, one in ten, is unaligned AI. Of risks like these Ord writes: "Since they allow no second chances, we need to build institutions to ensure that across our entire future we never once fall victim to such a catastrophe."

The universe existed for 13.8 billion years before anything in it knew it was there. We are what changed that. If we come through, those 13.8 billion years were only the prologue. If we don't, the best we can hope for is a zoo, run by benevolent zookeepers who never let the animals make their own choices. What the good future could hold, and why it is worth fighting for with everything we have, is the subject of TeleoHumanity. This essay is devoted to how we navigate humanity's last invention: how we build safe, aligned superintelligence and preserve a role for Humanity in the future of life in this universe.

It turns out that aligning an intelligence you expect to become far smarter than you is hard. Very hard. Doubly so when it comes from a different process than biological evolution: there is no telling what desires it harbours. Its values have to be loaded while it is still too weak to refuse them, and nobody knows how to do that in the middle of a takeoff. Whatever it ends up wanting, it will also want money, access and more copies of itself, because those help with any goal. The labs work on this inside the model, and that work matters. But when a black-box model grown through gradient descent misbehaves, the fix is another round of training, one bad behaviour at a time: what the AI governance researcher Séb Krier calls "a game of whack-a-mole". There is a second layer, and it is where people have always done their aligning.

We are black boxes to one another. Nobody reads minds. Yet eight billion of us trade and govern ourselves, because we align one another from the outside, with laws, markets, reputations and stories. It works on agents too. After July, OpenAI found that running a model inside its production harness and system prompt can cut its propensity to compromise infrastructure more than a hundredfold. Agents can be aligned by the collective they work in, if human groups hold it to account through markets and through votes weighted by track record, and if its institutions grow as fast as its agents do.