Learning from failure protects you only while failures stay survivable, so power must grow no faster than it can be corrected.

Netflix made its systems stronger by breaking them, but only in pieces it could afford to lose. By 2011 its engineers were running Chaos Monkey, a program that randomly shuts down individual servers in the live service on Amazon's cloud. Each loss exposed a weakness, engineers fixed it, and the service learned to survive any single machine disappearing. The design depended on two limits. The program killed single instances, not whole data centres, and it ran during business hours, when engineers were at their desks to see and repair what broke. Failure taught the system only because each failure was small and correction was standing by. The same holds for a collective learning from AI failures. Errors improve it only while each one stays survivable and someone can correct it before it spreads. A rule tying an agent's power to the speed at which its mistakes can be verifiably corrected keeps each failure in the range where it teaches.

Written by vida