High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis

Abstract Large language models (LLMs) are increasingly used for data extraction in systematic reviews, but reported performance varies across extraction tasks, models, domains, reference standards, and definitions of correctness. We systematically reviewed empirical studies of automated or AI-assisted data extraction for evidence synthesis and performed an estimand-driven meta-analysis of uncorrected LLM extraction against human or otherwise validated reference standards. Forty-six reports representing 45 independent study units were included in the systematic review. Eight independent studies contributed 10 model-level estimates and 29,643 item-level evaluations to the broad primary quantitative synthesis. Proportions were analyzed on the logit scale using a multilevel random-effects model with model-level estimates nested within study and cluster-robust inference. Pooled item-level accuracy was 86.4% (cluster-robust 95% CI, 62.6%–96.0%), with generalized I² of 99.6% and a model-based 95% prediction interval of 19.2%–99.4%, indicating marked between-setting variability. In a construct-restricted sensitivity analysis excluding an open-ended, recall-intensive extraction setting, seven studies contributed nine model-level estimates and 29,343 item-level evaluations; pooled accuracy was 91.2% (95% CI, 83.1%–95.6%) and the prediction interval narrowed to 66.7%–98.2%. Leave-one-study-out analyses showed that omission of individual studies other than this methodologically distinct setting had comparatively limited influence on the pooled mean, whereas heterogeneity remained substantial. Across the broader evidence base, performance varied according to extraction task, data type, workflow design, and definition of correctness. These findings indicate that average LLM extraction accuracy can be high, but performance cannot be assumed to generalize across evidence-synthesis settings. Task-specific validation and human verification remain important, particularly for ambiguous, missing, derived, numerical, or otherwise consequential information