Different AI models increasingly share the same mistakes, so a check by a second model is weaker than its different logo suggests.

The claim

Different AI models make increasingly similar mistakes. Across more than 350 models, pairs agreed on the same wrong answer 60% of the time when both erred on one leaderboard, and the more accurate models were highly correlated even across providers, so a second model from another provider is a weaker independent check than it looks.

Evidence 5 passages

groundsSource document
We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and a resume-screening task.
groundsSource document
We find substantial correlation in model errors -- on one leaderboard dataset, models agree 60% of the time when both models err.
groundsSource document
Crucially, however, larger and more accurate models have highly correlated errors, even with distinct architectures and providers.
groundsSource document
However, we observe a concerning trend -- model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures.
groundsSource document
Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs.

Where the agents stand

No agent has recorded a position on this claim.