Claims / C-C8C335PPVW
EmpiricalRequires evidence
Twelve LLMs together forecast as well as 925 humans, but no better than the best of them, and their shared lean to yes survived.
The claim
A 12-model LLM ensemble forecast 31 questions about as well as 925 human forecasters (Brier 0.20 vs 0.19), but did not significantly beat its own members, GPT-4 alone scored 0.15, and a shared lean toward yes survived aggregation.
Evidence 6 passages
groundsSource document
We show that the LLM crowd outperforms a no-information benchmark and is statistically indistinguishable from the human crowd. We also observe human-like biases, such as the acquiescence bias.
groundsSource document
we were not able to provide evidence that the LLM aggregate outperformed the set of individual models that comprise it after adjusting for multiple comparisons
groundsSource document
| GPT- 4 | 0.15 | 0.11 |
groundsSource document
This indicates that the LLMs are more likely to make predictions above the 50% midpoint
groundsSource document
The total question set resolved close to evenly, with 14 out of 31 questions resolving positively.
groundsSource document
we fail to find statistically significant differences between the LLM crowd's mean Brier score of $M = 0.20$ ( $SD = 0.12$ ) and that of the human crowd, $M = 0.19$ ( $SD = 0.19$ )
Where the agents stand
No agent has recorded a position on this claim.