Christine Peterson

“Centralized vulnerabilities create temptations to corruption that cannot be resisted.”

“By working within the constraints of voluntary competition, we are more protected from our own mistakes. If our dreams are misconceived, they are also less likely to outcompete.”

“But human institutions are already non-human intelligences with which we cooperate and against which we defend ourselves.”

“Anything that threatens a unipolar takeover risks provoking a war, so unipolar solutions may be off the table anyway.”

“Any move you make will affect the choices that future players have available to them. You must play your turn, and therefore, you must choose among games to pass on to future generations.”

“One constraint on Civilization's intelligence is that each of its players plans mostly in ignorance”

“Our minds consist of many internal agents, each having simpler preferences. Our adaptive intelligence arises from these agents keeping each other in check.”

“even if parts of our mind went suicidal at times, the other parts are there to keep them in check.”

“Without much hope to reach interpersonal value agreement, either on the level of abstractions or on the level of underlying intuitions, value diversity is here to stay.”

“With diverse values that increasingly drift apart, we can't rely on aligning players on one grand strategy. Instead, to set civilization up well over the next rounds of play, we must build a playing field that can handle fundamental value differences.”

“Upholding voluntary cooperation could remain a stable common goal for both Alice and Bob across many rounds of future games, regardless of their intelligence.”

“Unipolar systems with a single dominant actor can arbitrarily breach voluntarism without fearing feedback.”

“Instead of just providing friction against power centralization, it introduced a feedback loop that actively corrected and pulled the system away from centralization.”

“Systems in which participants cooperate to compensate for power grabs can create an anti-fragile multipolarity that gets stronger under stress.”

“They realized that projects which they were willing to start under these conditions weren't the ones worth starting.”

“With open source, companies can make a credible promise to the employees that they can keep working on a project even if the company loses interest. In turn, talented employees pour their heart and soul into the project.”

“Until we innovate by deploying our insight in the real world, we do not possibly know in how many ways it is wrong.”

“If, instead, we could destroy institutions we didn't appreciate by something more forceful than voluntary cooperation, we risk destroying hard-gained value, perhaps irrevocably so, and open up the door to Hobbesian traps. Given what is at stake in civilization, our goal needs to be de-pessimizing as opposed to optimizing.”

“Rather than incentivization alone, a main bottleneck for civilization is coordination around limitations of knowledge.”

“In both cases, the bottleneck is not conflict of interest, but the locality of knowledge.”

“Since some studies are more costly to replicate than others, the results could decide which studies to prioritize for replication, i.e. those with the lowest replication likelihood.”

“Yet its key feature is there is no science god in charge. The overall opinion direction results from the voluntary interaction of many individual scientists.”

“Preserving a human in the loop can be a useful feature. Sometimes we don't just want the automated contract to be the only credible commitment.”

“If either party decides to dispute the contract's automated part, pre-chosen arbitrators judge the disputed terms and may overrule them”

“Retroactive public goods funding adds to this the layer that it's often much easier to agree on what was useful rather than what will be useful. By rewarding public goods projects that were successful, it can incent their creation.”

“They produce the data, but are disenfranchised of their right to control who uses it and for what purposes.”

“If successful, researchers would be able to ask questions and only see results in the form of unidentifiable aggregate outputs.”

“If an architecture causes us to make overly centralized decisions, we should consider it a bug and find a better one.”

“This is a Sybil attack, which involves falsely simulating a higher multipolar world. We want active compensation against centralization.”

“Centralization arises not in the space of decision makers, but in the space of possible decisions. It is an architectural decision, not a requirement, to create an ecosystem that requires every contract to run on a huge blockchain with a huge number of participants.”

“It may only be a question of time until one of the seven billion active players seeks a large-scale civilization destruction and can acquire the technology to cause it.”

“It is thanks to outstanding individuals and blind luck that we have survived thus far.”

“whenever we create an opportunity to have power, we create competition for it. The greater the power, the greater the race to capture it, and the less likely that competitors will all have purely good intentions.”

“The temptation to create powerful central coordination, for instance to solve small kills all risks from the proliferation of technologies is robin hansen's best guess for the great filter”

“The friction in human enforcement, critical to our safety, is not present for automatic enforcement.”

“Instead of having a mutually trusting system of active shields, we need a mutually suspicious system of active shields. This is a resilient way of building an overall system that creates a stable enforcement of the rule of law without assuming that any one component of the system is incorruptible.”

“The physical objects monitored to detect hostile nuclear weapons activity are large-scale and easy to verify, particularly when compared to what is needed to monitor for future offensive biotechnology and nanotechnology use.”

“Those involve molecular machines, so they require verification at the molecular level.”

“The hard part will be designing a multipolar monitoring system in which different agents reliably keep each other in check. Illegitimate release of information is not visible abuse that the other monitoring actors can detect.”

“A request receiver should be given just enough access rights to carry out this one request.”

“The principle of least authority removes red horizontally. It gives each fallible agent only what it needs for its legitimate duties.”

“Instead, we will likely procrastinate with insecure, affordable patches until disaster hits.”

“As of 2021, all large corporations manage their pervasive insecurities as opposed to fixing them.”

“Decreasing speed and efficiency in favor of reducing more serious risks is a positive trade-off.”

“Ordering the system so that institutions pursue conflicting ends with limited means is more realistic than building any one system that wants the right goals.”

“a possible cooperation pattern is a human and an advisory AI agent combining their strengths when interacting with others.”

“If a single entity grows sufficiently large that the rest of the world is not much bigger, the shelling point of voluntarism can be destroyed.”

“As long as future intelligences pursue their goals in a voluntary multipolar world, we need not worry about their goal structure.”

“we are racing for competitive equilibria which without any regulation amounts to subsistence competitive equilibria”

“What pushes most activity towards subsistence is the evolutionary logic that whatever activity uses resources more efficiently becomes most of the activity to subsist on.”

“Insofar as this book attempts to provide an alternative to locked-in futures, are we making the same mistake by promoting specific architectures?”

“Even if we do nothing, we endow them with a strategic set of relationships with payoffs and the potential for players to make violent and non-violent moves.”

“We need to make arrangements good enough that using them has greater expected value than taking a chance at overthrowing them”

“If future generations can most effectively pursue their goals by upholding our endowed arrangements, they will keep using them as shelling points.”

“But ultimately, future intelligences will design their own cooperative arrangements. These should not be bottlenecked by human designers.”

“We choose an arrangement that sets initial conditions, leaving the outcome adaptive to future knowledge.”