Michelle Vaccaro

“As shown in Fig. [2,](#page-3-0) when the human alone outperformed the AI alone, the combined human–AI system outperformed both alone with an average pooled effect size for human–AI synergy of g = 0.46 (t104 = 5.06; two-tailed P = 0.000; 95% CI, 0.28 to 0.66), a medium-sized effect[21](#page-8-10). But when the AI alone outperformed the human alone, performance losses occurred in the combined system relative to the AI alone, with a negative effect size for human–AI synergy of g = −0.54 (t104 = −6.20; two-tailed P = 0.000; 95% CI, −0.71 to −0.37), a medium-sized effect[21.”

“With most of the decision tasks studied in our sample, however, both the human and the AI system make a complete decision, with the humans usually making the final choice. Our results suggest that with these decision tasks, better results might have been obtained if the experimenters had designed processes in which the AI systems did only the parts of the task for which they were clearly better than humans. Only 3 of the 100+ experiments in our analysis explore such processes with a predetermined delegation of separate subtasks to humans and AI.”

“Only 3 of the 100+ experiments in our analysis explore such processes with a predetermined delegation of separate subtasks to humans and AI. With the four effect sizes from these 3 experiments, we found that, on average, human–AI synergy (g = 0.22, t104 = 0.69; two-tailed P = 0.494; 95% CI, −0.42 to 0.87) occurred, but the result was not statistically significant (see Supplementary Information section 2.6 for a more detailed discussion of these experiments).”

“Most (>95%) of the human–AI systems in our dataset involved humans making the final decisions after receiving input from AI algorithms. In these cases, one potential explanation of our result is that, when the humans are better than the algorithms overall, they are also better at deciding in which cases to trust their own opinions and in which to rely more on the algorithm's opinions.”

“In this case, we hypothesize that, since the people were less accurate, in general, than the AI algorithms, they were also not good at deciding when to trust the algorithms and when to trust their own judgement, so their participation resulted in lower overall performance than for the AI algorithm alone.”

“In our broad sample of recent experiments, the vast majority (about 85%) of the effect sizes were for decision-making tasks in which participants chose among a predefined set of options. But in these cases we found that the average effect size for human–AI synergy was significantly negative.”