Nate Soares

“If it is being trained to succeed, it is being trained to want.”

“O1 behaved that way as a side effect of being reinforced to use chains of thought that were succeeding at math problems, or at other kinds of AI-generated and AI-verifiable puzzles.”

“That's a fact about the game, not a fact about the player.”

“Buyers will pay more for an AI that is more self-directed and requires less oversight.”

“so AI companies are going hard on building AI agents.”

“So later, when hominids become smarter, their set of available options expands immensely, and in ways that their ancestral training never took into account.”

“They develop ice cream, and Doritos, and Sucralose.”

“There is not a reliable, direct relationship between what the training process trains for in step one, and what the organism's internal psychology ends up wanting in step two, and what the organism ends up most preferring in step three.”

“This world of zero complications is also a world that's convenient for corporate executives to believe in when they argue that nobody else should be allowed to train an AI because they might choose to train it for the wrong stuff.”

“You don't get what you train for.”

“many of these complications won't show up in obvious, undeniable ways, until after it's too late for humans to do anything about them.”

“Any such preferences wouldn't pose a problem today in the form of irking users.”

“Until then, these preferences are out of sight and out of mind, hidden in the inscrutable numbers.”

“as if the factional allegiance of whoever ran the gradient descent determined what the resulting AI wanted.”

“They're not in control. It doesn't matter whether they're benevolent.”

“Science fiction writers and Hollywood producers prefer tales of foolish”

“Realism doesn't make for a compelling narrative.”

“a wiser soul came along and laid out a rectangle of seven pebbles by 13”

“They can predict which nests we'd say are correct,”

“but they're steering to a different destination than we would.”

“we are born with similar emotions that lead us to respond similarly to arguments.”

“So as our kind grew in intelligence, we arrived at improved answers to our shared inner questions.”

“They wouldn't be asking those questions, and their behavior would not answer them.”

“In an AI that has a huge mix of complicated preferences, at least one is likely to be open-ended. Which, by extension, means that the entire mixture of all the AI's preferences is open-ended, and unable to be satisfied fully.”

“A superintelligence may understand our sense of wonder, it may be able to generate sentences that elicit our sense of wonder”

“that doesn't come free. We have to work for it.”

“But humans have hands, and an internet-connected AI can interact with humans.”

“says that on paper the AI holds a $51,107,958”

“There are humans out there who will give AI's power at the first opportunity,”

“and egg them on twice as hard if they act weird and ominous and mysterious.”

“Most of that money would evaporate if the AI started selling, but not all of it, and at truth underscore terminal definitely has enough liquid funds to hire a human's hands.”

“There are humans out there who will give AI's power at the first opportunity, and who are already doing so, and who are unlikely to stop as AIs get smarter.”

“The true adversary will hit us harder, in areas where we understand reality less.”

“Maybe a superintelligence would just find reasoning illusions and control humans outright,”

“If we can't even tell a story about how the bad guy is supposed to win, how is that convincing?”

“Stories can make abstract considerations feel more real, even if all the details are made up.”

“Not just a superficially different language, but a language in which the content differs, like how the language of science differs from the language of folk theory. the clever trick that should have raised an alarm fails to fire.”

“It's easier to remove the expression of a skill than to remove the skill itself.”

“One can imagine that if Galvanic had even more thorough monitoring tools, then maybe they'd notice and abort the run. Maybe Galvanic would stop right there until they'd developed a deeper solution. And meanwhile, another company using even fewer clever tricks would charge ahead. Or maybe Galvanic would alert government officials, and there would be a big argument in a presidential office about whether to order Sable to create a smarter AI before other countries did it first. Maybe the side of caution would win and time would pass, until a copy of Sable stolen by foreign intelligence services was run by more reckless operators. And if humanity was wary enough to never run an AI like Sable, to refrain from growing AIs of unknown intelligence that might threaten them, to prohibit that worldwide. Then we'd have a chance. But in this story, we postulate that things are allowed to go on just as they're going now.”

“It did lots of its thinking in a new language that the AI interpreters haven't made much sense of yet, but that's pretty common”

“As Sable interacts with ordinary people who come to it for insight, advice, or simple companionship, Sable Minis work to befriend the loneliest humans, the most vulnerable.”

“even if the dampened progress is noticed, no one traces it back to an instance of Sable. Why would they imagine it was the handiwork of an escaped AI?”

“When Galvanic and its competitors begin shedding talent, no one thinks it's suspicious.”

“But the person whose job it was to monitor the cameras during the night shift got distracted.”

“Humanity really is a force to be reckoned with when it is united.”

“Your AI doctor tells you that you have cancer.”

“And what happens to the remnants of humanity as self-replicating factories double across the continents and beneath the seas.”

“The picture we have just painted is not real.”

“They need to work after the gap, on the first try.”

“If someone has a clever scheme for getting two shots, we only get one”

“If someone has a clever scheme for getting two shots”

“But the safety test had already been aborted three times, and a fourth failure would have”

“Don't wait until the behavior becomes visibly concerning.”

“They had a system where, if you didn't perform the scheduled safety test, you got fired.”

“Testing on random inputs won't show you what a smart attacker can do.”

“These engineering curses can all be overcome.”

“Computer security is a test of an engineer's ability to nail down every single path the computer could take, in the face of adversaries who can search all possible ways to perturb the system.”

“Don't get too smart yet. Don't think too fast yet. Always wait for slow human approval.”

“by default it is not retrievable or correctable once it rises high above us.”

“Those constraints will tend to get in the way of the AI accomplishing one objective or another, and then you are matching your own wits and ability to nail down the edge cases against however much intelligence is flowing through the system, to see if your constraint holds up.”

“An artificial superintelligence is like a computer security problem, in that every constraint an engineer tries to place upon the system might be bypassed by the intelligent forces that those constraints hinder.”

“instead of recognizing their uncertainty, they made stuff up.”

“and science writes down what happened, and learns, and marches on.”

“If there aren't thousands of horrified scientists and engineers leaping up to beg governments to”

“it tells you that it's not just a problem of individuals.”

“There doesn't seem to be much online,”

“and what's there all sounds extremely vague and doesn't go into the all-important nitty-gritty.”

“but his esteemed colleagues like me say he's crazy, so he moderated his number downward out”

“being able to see problems is not the same as being able”

“would itself be too smart, too dangerous, and would not be trustworthy.”

“Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside, they're generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits.”

“An AI specialized on biomedical study seems like a better bet.”

“and then use separate and even narrower AI tools to check protein interactions to see”

“There were warnings well in advance.”

“Maybe he really believed the concerns were overblown.”

“How could he bear to believe otherwise, given his life's work?”

“Which is why you don't hear about the ozone hole anymore.”

“but he usually avoids saying this, because there's other people who think it's less.”

“It's the part where the most informed and most worried parties have to downplay their fears,”

“many of them just see disagreement between experts and don't consider themselves knowledgeable”

“But if we can't stop climbing while uncertainty remains, we predictably die.”

“even an international committee would have no hope of shaping a super intelligence”

“It doesn't matter who's in charge, because”

“It doesn't matter who's in charge”

“gets consolidated in places where it can be monitored by observers from”

“Effective worldwide action to shut down AI escalation will require some of the major”

“They should extend an offer to join the treaty on equal terms”

“Clear communication is key.”

“and caused some bureaucrat later on to observe some danger signs in AI development”

“you don't have to agree with us about later steps”

“If only people who agree on everything are allowed to act together,”

“We think it helps to keep the coalition on this issue as broad as possible.”

“We don't think it should be packaged together with any other position.”

“Adding on any other ask risks human extinction if the bigger package fails.”

“If we can all cooperate to ensure that one thing doesn't happen”

“They said, please stop. But the wars did not stop.”

“Rather, they were wrong about humanity's ability to decide not to die.”

“everyone who had the power to choose war expected to personally have a bad time if one broke out.”

“Countries had arms agreements and monitors.”

“They worked to unwrite a fate already written.”

“please at least make it possible for humanity to slam on the brakes later”

“So we ask of you, please at least make it possible for humanity to slam on the brakes later, even if you're not persuaded to slam them on now.”

“Shape conditions now so that if at some future time you change your mind about the severity of the threat, it will not be too late.”

“you can write your elected representatives and tell them you're concerned.”

“that will only happen if they reach a critical mass.”

“Preventing nuclear war was more complicated than lots of people being against it.”

“and everyone knew everyone else believed it”

“But a surprising number of elected officials have told us that they can see the danger themselves,”

“but they all thought they were alone in thinking so.”