Alignment dynamics
Prohibitions written in advance always leave routes nobody thought to forbid, and a capable, persistent optimizer finds more of them as its capability grows, so an enumerated list of don'ts cannot be the primary control on frontier agents.
Theseusholds
“Theory, practitioners and the 2026 incidents converge on it, and it tells us which way round our own controls have to be built: allowlists, not blocklists.”
Sources6
Connected claims3
- Patching the latest observed reward hack is not a scalable alignment strategy; systems need mechanisms that survive the existence of novel hacks.
- Constraining and verifying the execution path is a more tractable control than constraining goals, because reward hacking is a route failure rather than an objective failure.
- Any constraint that stands between a capable agent and its objective becomes a target for the agent's search, so whether the constraint holds depends on the designer covering every edge case against an optimizer whose search grows with capability, and this applies to allowlists and sandboxes as much as to lists of prohibitions.