The increasing integration of artificial intelligence (AI) into various aspects of human life, from professional fields like medicine and finance to daily activities, has sparked significant research into human-AI collaboration. The potential for synergy between human intelligence (reasoning, creativity) and AI's computational power (speed, scalability) is substantial. While some studies highlight the benefits of this collaboration, others demonstrate that human-AI systems don't always surpass the performance of either humans or AI alone. This discrepancy necessitates a comprehensive understanding of the conditions under which human-AI combinations are most effective. This study aims to address this critical question by quantitatively assessing the synergy and augmentation effects of human-AI systems across a range of tasks and contexts.

The researchers conducted a thorough literature review focusing on studies that directly compared human-only, AI-only, and human-AI system performance. The search spanned interdisciplinary databases (ACM Digital Library, Web of Science, AIS eLibrary) from January 2020 to June 2023. Inclusion criteria required original experiments with quantitative performance measures for all three conditions. The review aimed to address the gap in understanding human-AI synergy, going beyond the common focus on human augmentation (where the human-AI system outperforms humans alone) to evaluate true synergy (where the combined system outperforms both individual components). Existing literature reveals mixed findings, necessitating this quantitative analysis to identify factors influencing human-AI effectiveness.

The meta-analysis included 370 effect sizes from 106 unique experiments, extracted from 74 eligible papers identified through a multi-stage search and screening process. Hedges' g was used to calculate the standardized mean difference in performance across human-only, AI-only, and human-AI conditions. A three-level meta-analytic model was employed to account for the non-independence of effect sizes within and across experiments, with robust variance estimation methods applied to handle potential dependencies in the data. Moderator analyses examined several factors, including task type (decision vs. creation), relative performance of humans and AI, type of AI, year of publication, experimental design, inclusion of AI explanations or confidence levels, and participant type. Publication bias was assessed using funnel plots, Egger's regression, and rank correlation tests. Sensitivity analyses were performed to evaluate robustness to outliers and influential data points.

The primary analysis revealed a significant lack of human-AI synergy. On average, human-AI systems performed worse than the best of human-only or AI-only systems (Hedges' g = -0.23, 95% CI: -0.39 to -0.07). However, there was substantial evidence of human augmentation, with human-AI systems consistently outperforming humans alone (Hedges' g = 0.64, 95% CI: 0.53 to 0.74). Moderator analyses revealed significant effects for several factors. Task type was a significant moderator; decision tasks exhibited performance losses (g = -0.27) while creation tasks suggested potential gains (g = 0.19), although not statistically significant due to a small sample size. Crucially, the relative performance of humans and AI alone was a strong predictor of human-AI synergy; human-AI systems outperformed both when humans initially outperformed AI (g = 0.46), but underperformed when AI was initially superior (g = -0.54). The type of AI, year of publication, and experimental design also had minor but statistically significant effects. Factors such as explanations, confidence levels, and participant types were surprisingly non-significant moderators.

The finding of a lack of overall human-AI synergy highlights the importance of careful system design and task allocation. The results suggest that simple averaging of human and AI performance is inaccurate and misleading. The significant interaction between the relative performance of humans and AI alone suggests a complex interplay where effective human-AI collaboration depends heavily on exploiting the strengths of each partner. The significant effect of task type further indicates the necessity of tailoring human-AI designs to the specifics of the task; generative AI approaches are particularly promising for creation tasks. The surprisingly insignificant roles of AI explanations and confidence levels challenge the prevailing emphasis on these factors in previous studies, redirecting attention towards task allocation and relative capabilities as more critical design considerations.

This meta-analysis demonstrates that human-AI systems, while often augmenting human performance, frequently fall short of achieving synergy. The results suggest future research should concentrate on developing effective methods for integrating humans and AI, particularly through task-specific designs that leverage individual strengths and creative process design, especially with generative AI for creation tasks. Further research should also focus on developing more robust evaluation metrics that encompass multiple performance criteria and creating standardized reporting guidelines to improve comparability and facilitate cumulative progress in the field. These developments are key to unlocking the true potential of human-AI collaboration.

The meta-analysis is limited by the availability of studies meeting the inclusion criteria, which could represent a bias in the published literature. The variability in study designs, participant populations, and performance metrics also limits direct comparability across studies. While publication bias tests indicated no significant bias overall, some asymmetry in human augmentation results was detected. The analysis relies on the quality of the individual studies included and might not fully capture the diverse ways humans and AI are used in practice outside of controlled experiments. Finally, unexplained heterogeneity remains, suggesting potential unmeasured moderators that require further investigation.