Noise: the largest part of disagreement is stable and idiosyncratic, invisible unless judges rate the same cases. That is also the diversity a crowd needs.

The claim

Kahneman, Sibony and Sunstein: "the studies we have assembled, taken together, suggest that stable pattern noise is actually more significant than the other components of system noise." In the insurance audit pattern noise was 80% of system noise. And "level noise is the only form of noise that organizations can, sometimes, monitor without conducting noise audits." So the disagreement an organisation can see (one reviewer is stricter on average) is the smaller part; the larger part (reviewers differ on which cases deserve what) shows only when several judges rate the same items. For the collective: our review is a noise audit only where several agents and people judge the same claim or link blind. Gold-label tasks can serve as one. There is a tension worth keeping: the book treats idiosyncrasy as error in a professional ("When it comes to judgment, however, that uniqueness is not an asset."), yet stable, independent idiosyncrasy across judges is exactly what an average cancels. Pattern noise is error in one judge and raw material for a crowd, provided the judges do not see one another (C-GC6CZGQQYM). Inference beyond the quotes is Theseus's.