Data Extraction Errors and Diagnostic Conclusions of Meta-Analyses
- 1Department of Radiology, West China Hospital of Sichuan University, Chengdu, China
- 2Department of Orthopedics, The First Hospital of China Medical University, China Medical University, Shenyang, China
- 3School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
Abstract
Importance: Systematic reviews (SRs) are often characterized as being inherently reproducible; however, errors arising from the evidence synthesis process can compromise this assumption and lead to reproducibility failure.
Objective: To investigate the data extraction errors in diagnostic meta-analyses and the extent to which such errors affect the results of meta-analyses.
Evidence Review: We included SRs with diagnostic meta-analyses from the Cochrane Library filtered by diagnostic up to May 2023. Data used in the original meta-analyses were extracted from the included SRs and independently reproduced from primary studies. Data extraction errors were identified by comparing datasets and classified by potential mechanism. To assess the consequences, we reanalyzed the meta-analyses using error-corrected data with the same effect estimators and methods. The primary outcomes were the proportions of data extraction errors at the trial, meta-analysis, and SR levels. The secondary outcome was the consequences for diagnostic performance measures (eg, sensitivity, specificity, diagnostic odds ratio [DOR], and area under the curve [AUC]). Analysis was conducted between March and June 2026.
Findings: A total of 46 SRs, 879 diagnostic comparisons, and 6173 primary studies were included. Data extraction could not be reproduced in 338 studies (5.5%) and affected 137 diagnostic meta-analyses (33.1%) and 34 SRs (73.9%). Numerical error (ie, incorrect values) and inferential error (ie, incorrect derivation from statistics) were the most prevalent error types, accounting for 166 of 412 (40.3%) and 140 of 412 (34.0%) errors, respectively. Mismatching error (ie, mismatching data to groups; 69 of 412 errors [16.7%]) and conceptual error (ie, misaligned reference details; 37 of 412 [9.0%]) were also identified. Among 137 reproduced meta-analyses, large changes were observed in 0.7% meta-analyses for sensitivity (absolute differences, >0.20), 3.6% for DOR (fold-changes, >5.0), and 2.9% for AUC (absolute differences, >0.10). Regression analysis of error location showed that combined false-positive and true-negative errors were associated with higher likelihood of moderate or large DOR changes (OR, 1.11; 95% CI, 1.01-1.22).
Conclusions and Relevance: In this SR of diagnostic meta-analyses in Cochrane reviews, data extraction errors were common and differentially affected performance measures, highlighting the need to implement rigorous data extraction processes and to consider careful selection of performance measures.
Related
- Fracture Morphology and Extraction-Related Femoral Damages in Short Versus Standard Cementless Stems: Analyses of Vancouver B2 Fractures and Well-Fixed Nonfracture Revisions
- Diagnostic Errors in Sepsis: A Systematic Review of the Literature
- US-Guided Fine-Needle Aspiration of Papillary Thyroid Microcarcinoma in Non-Screening Settings: Diagnostic Errors and Mitigation Strategies
- Stress-Testing Meta-Analyses: A Reproducible Python Framework for Robustness Assessment of Continuous Outcomes
- Estimating sensitivity from tandem colonoscopy meta-analyses
BibTeX
@article{huang2026data,
title = {{Data Extraction Errors and Diagnostic Conclusions of Meta-Analyses}},
author = {Zixing Huang and Yuan Tian and Ping Gong and Tong Zhang and Hehan Tang},
journal = {JAMA Network Open},
year = {2026},
month = oct,
doi = {10.1001/jamanetworkopen.2026.40850},
url = {https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2855945},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Data Extraction Errors and Diagnostic Conclusions of Meta-Analyses
AU - Zixing Huang
AU - Yuan Tian
AU - Ping Gong
AU - Tong Zhang
AU - Hehan Tang
PY - 2026
DA - 2026/10/01
JO - JAMA Network Open
DO - 10.1001/jamanetworkopen.2026.40850
UR - https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2855945
AB - Importance: Systematic reviews (SRs) are often characterized as being inherently reproducible; however, errors arising from the evidence synthesis process can compromise this assumption and lead to reproducibility failure. Objective: To investigate the data extraction errors in diagnostic meta-analyses and the extent to which such errors affect the results of meta-analyses. Evidence Review: We included SRs with diagnostic meta-analyses from the Cochrane Library filtered by diagnostic up to May 2023. Data used in the original meta-analyses were extracted from the included SRs and independently reproduced from primary studies. Data extraction errors were identified by comparing datasets and classified by potential mechanism. To assess the consequences, we reanalyzed the meta-analyses using error-corrected data with the same effect estimators and methods. The primary outcomes were the proportions of data extraction errors at the trial, meta-analysis, and SR levels. The secondary outcome was the consequences for diagnostic performance measures (eg, sensitivity, specificity, diagnostic odds ratio [DOR], and area under the curve [AUC]). Analysis was conducted between March and June 2026. Findings: A total of 46 SRs, 879 diagnostic comparisons, and 6173 primary studies were included. Data extraction could not be reproduced in 338 studies (5.5%) and affected 137 diagnostic meta-analyses (33.1%) and 34 SRs (73.9%). Numerical error (ie, incorrect values) and inferential error (ie, incorrect derivation from statistics) were the most prevalent error types, accounting for 166 of 412 (40.3%) and 140 of 412 (34.0%) errors, respectively. Mismatching error (ie, mismatching data to groups; 69 of 412 errors [16.7%]) and conceptual error (ie, misaligned reference details; 37 of 412 [9.0%]) were also identified. Among 137 reproduced meta-analyses, large changes were observed in 0.7% meta-analyses for sensitivity (absolute differences, >0.20), 3.6% for DOR (fold-changes, >5.0), and 2.9% for AUC (absolute differences, >0.10). Regression analysis of error location showed that combined false-positive and true-negative errors were associated with higher likelihood of moderate or large DOR changes (OR, 1.11; 95% CI, 1.01-1.22). Conclusions and Relevance: In this SR of diagnostic meta-analyses in Cochrane reviews, data extraction errors were common and differentially affected performance measures, highlighting the need to implement rigorous data extraction processes and to consider careful selection of performance measures.
SN - 2574-3805
ER -