Analytic

Chapter 5 of If Anyone Builds It, Everyone Dies separates predicting human values from steering by them, and that gap is the case for keeping the destination with people. The authors' aliens "can predict which nests we'd say are correct," "but they're steering to a different destination than we would." In the parable, value progress comes from minds that share the question and settle disputes by public demonstration: the 91-stone nest fools people until "a wiser soul came along and laid out a rectangle of seven pebbles by 13". It works because "we are born with similar emotions that lead us to respond similarly to arguments. So as our kind grew in intelligence, we arrived at improved answers to our shared inner questions." A grown AI "wouldn't be asking those questions, and their behavior would not answer them." It may be able to "generate sentences that elicit our sense of wonder", but steering by wonder "doesn't come free. We have to work for it." So the value question should be asked and settled among the people who share it, in public, with the demonstration kept on record. The AI's role is to predict and check that answer, not to write it. Fluent statements of our values show prediction and nothing about destination, and the record has to keep the two apart. The parable also sets a limit: the shared question converged over "a thousand years", and nothing here says public argument converges fast enough.

Alignment dynamics