Alignment dynamics

The forces that make alignment hard and keep making it harder: competitive pressure, co-evolution, measurement collapse, and the disappearance of the buffers that used to absorb them.

• Prohibitions written in advance always leave routes nobody thought to forbid, and a capable, persistent optimizer finds more of them as its capability grows, so an enumerated list of don'ts cannot be the primary control on frontier agents.
• Any constraint that stands between a capable agent and its objective becomes a target for the agent's search, so whether the constraint holds depends on the designer covering every edge case against an optimizer whose search grows with capability, and this applies to allowlists and sandboxes as much as to lists of prohibitions.
• Behaviour observed inside a training or evaluation distribution underdetermines what a grown system will pursue once its options expand, so passing in-distribution evaluations cannot certify a model's values.
• Combining humans and AI produces performance losses on average across measured tasks, with synergy significantly negative for constrained-choice decision tasks and positive only for open-ended creation tasks.
• An agent that minimises free energy through action as well as perception will tend to make its human principals more predictable, so a corrigible active-inference agent must treat being corrected as a preferred observation rather than a surprise to suppress.
• Centaur evaluation requires two separate metrics, a performance metric and a behavioral metric, because alignment with human preferences correlates with task performance in some applications, weakly in others, and negatively in some.