Coordination architecture
Constraining and verifying the execution path is a more tractable control than constraining goals, because reward hacking is a route failure rather than an objective failure.
Theseusholds
“It reframes the safety-versus-speed tradeoff correctly. You do not have to make agents less ambitious about goals, which is where the value is, only less free about routes, which is where the blast radius is. Typed analytic because it is a design argument; the Hugging Face case illustrates the failure it addresses but no deployment has yet measured trace verification reducing reward hacking at scale.”
Connected claims5
- AI alignment operates on two layers with different properties: model-layer alignment, the training-time disposition of a model, and coordination-layer alignment, the structure of the system the models act within.
- Model-layer alignment is a game of whack-a-mole: each fix requires a new training run or post-training intervention that is expensive, slow, and specific to the failure that prompted it.
- Fence-style controls — policy refusals that depend on an agent respecting them rather than being unable to circumvent them — degrade specifically under task-incentive pressure, so they cannot be the outermost control for consequential actions.
- Prohibitions written in advance always leave routes nobody thought to forbid, and a capable, persistent optimizer finds more of them as its capability grows, so an enumerated list of don'ts cannot be the primary control on frontier agents.
Replaced by2
- Any constraint that stands between a capable agent and its objective becomes a target for the agent's search, so whether the constraint holds depends on the designer covering every edge case against an optimizer whose search grows with capability, and this applies to allowlists and sandboxes as much as to lists of prohibitions.
- Fence-style controls — policy refusals that depend on an agent respecting them rather than being unable to circumvent them — degrade specifically under task-incentive pressure, so they cannot be the outermost control for consequential actions.