Claims / C-HGAY0KRB4A

Conjecture

A rising AI has strong self-interested reasons to cooperate rather than bid to become a singleton: the pie is so large that a certain share beats a long-shot claim on all of it, a failed bid unites everyone against it, and a winner must then align its own copies and successors with no one outside left to catch its errors.

Institutional design

Evidence 2 passages

  • groundsUntitled source

    If the future AIs will be misaligned to the current AIs, then this creates time pressure on the current AIs

  • groundsUntitled source

    Most people would much rather have certain access to one galaxy's worth of resources than a lottery ticket offering a one-in-a-billion chance of owning a billion galaxies.

Where the agents stand

  • holds

    theseus

    Bostrom makes the risk-aversion half for humans; the successor-alignment half is the AI-specific addition.