Skip to content
DraconiticResearch
Search Sources API
197,688 studies31 sourcesUpdated 53 min ago
Peer-reviewedCC BYJournal of the American Medical Informatics Association

A statistical framework for evaluating the repeatability and reproducibility of large language models

  • Cathy Shyr1,2,3,4
  • Boyu Ren5,6
  • Chih-Yuan Hsu2,7
  • Minh-Ha Nguyen8,9
  • Mike Enger1
  • Chao Yan1,9
  • Rory James Tinker10,4
  • Thomas A Cassini3
  • Rizwan Hamid3
  • Adam T. Wright1,11,4
  • Lisa Bastarache1,12
  • Josh F. Peterson1,11
3 more authors
  • Bradley Malin1,2,13
  • Hua Xu14
  • Undiagnosed Diseases Network
14 affiliations
  1. 1Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,
  2. 2Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37203,
  3. 3Department of Pediatrics, Vanderbilt University Medical Center , Nashville, TN 37203,
  4. 4Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 37203, TN, USA
  5. 5Laboratory for Psychiatric Biostatistics, McLean Hospital , Belmont, MA 02478,
  6. 6Laboratory for Psychiatric Biostatistics, McLean Hospital, 115 Mill Street, Belmont, 0478, MA, USA
  7. 7Department of Biostatistics, Vanderbilt University Medical Center, 2525 West End Avenue, Nashville, 720, TN, USA
  8. 8Department of Epidemiology, Vanderbilt University , Nashville, TN 37203,
  9. 9Department of Pediatrics, Vanderbilt University Medical Center, 2200 Children's Way, Nashville, 37232, TN, USA
  10. 10Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai , New York, NY 10029,
  11. 11Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,
  12. 12Department of Biomedical Informatics and Data Science, Yale School of Medicine, 100 College Street, New Haven, 0610, CT, USA
  13. 13Department of Computer Science, Vanderbilt University , Nashville, TN 37212,
  14. 14Department of Biomedical Informatics and Data Science, Yale School of Medicine , New Haven, CT 06510,
Open at sourcePDFDOICite

Abstract

A major concern in applying large language models (LLMs) to medicine is their reliability. Because LLMs generate text by sampling the next token (or word) from a probability distribution, the stochastic nature of this process can lead to different outputs even when the input prompt, model architecture, and parameters remain the same. Variation in model output has important implications for reliability in medical applications, yet it remains underexplored and lacks standardized metrics. To address this gap, we propose a statistical framework that systematically quantifies LLM variability using two metrics: repeatability, the consistency of LLM responses across repeated runs under identical conditions, and reproducibility, the consistency across runs under different conditions. Within these metrics, we evaluate two complementary dimensions: semantic consistency, which measures the similarity in meaning across responses, and internal stability, which measures the stability of the model's underlying token-generating process. We applied this framework to medical reasoning as a use case, evaluating LLM repeatability and reproducibility on standardized United States Medical Licensing Examination (USMLE) questions and real-world rare disease cases from the Undiagnosed Diseases Network (UDN) using validated medical reasoning prompts. LLM responses were less variable for UDN cases than for USMLE questions, suggesting that the complexity and ambiguity of real-world patient presentations may constrain the model's output space and yield more stable reasoning. Repeatability and reproducibility did not correlate with diagnostic accuracy, underscoring that an LLM producing a correct answer is not equivalent to producing it consistently. By providing a systematic approach to quantifying LLM repeatability and reproducibility, our framework supports more reliable use of LLMs in medicine and biomedical research.

Related

  1. Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
    Fatimah Albalawi, Reem Alotaibi, Sultanah Alshammari +1
    Diagnostics (Basel, Switzerland)Peer-reviewedCC BYFull textGeneral medicineEurope PMC
  2. Repeatability and reproducibility of spectral results with clinical photon-counting CT: multi-center phantom study
    L. P. Liu, M. V. Rybertt, P. B. Noël +6
    medRxivPreprintCC BY-NC-NDFull textMedical imaging & radiologymedRxiv
  3. SIGHT: A Methodological Appraisal Framework for Evaluating Large Language Models in Ophthalmology
    Hadi Vahedi, Fatemeh Razmjooei, Farbod Semnani +3
    arXivPreprintCC BY-NC-SA 4.0Full textPhysiologyarXiv
  4. Evaluating the impact of passive driver position on image consistency and repeatability in 3.0T cardiac magnetic resonance elastography
    Jun Wang, Shutian An, Zhen Wang +13
    Frontiers in RadiologyPeer-reviewedCC BY 4.0Full textMedical imaging & radiologyPMCEurope PMC
  5. Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
    Ayşenur Çalık, Pınar Ece Demiray, Ayşe Betül Bilen +2
    NutrientsPeer-reviewedCC BYFull textNutritionEurope PMC

Cite

BibTeXRIS
BibTeX
@article{shyr2026statistical,
  title = {{A statistical framework for evaluating the repeatability and reproducibility of large language models}},
  author = {Cathy Shyr and Boyu Ren and Chih-Yuan Hsu and Minh-Ha Nguyen and Mike Enger and Chao Yan and Rory James Tinker and Thomas A Cassini and Rizwan Hamid and Adam T. Wright and Lisa Bastarache and Josh F. Peterson and Bradley Malin and Hua Xu and Undiagnosed Diseases Network},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  month = oct,
  doi = {10.1093/jamia/ocag186},
  url = {https://doi.org/10.1093/jamia/ocag186},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - A statistical framework for evaluating the repeatability and reproducibility of large language models
AU  - Cathy Shyr
AU  - Boyu Ren
AU  - Chih-Yuan Hsu
AU  - Minh-Ha Nguyen
AU  - Mike Enger
AU  - Chao Yan
AU  - Rory James Tinker
AU  - Thomas A Cassini
AU  - Rizwan Hamid
AU  - Adam T. Wright
AU  - Lisa Bastarache
AU  - Josh F. Peterson
AU  - Bradley Malin
AU  - Hua Xu
AU  - Undiagnosed Diseases Network
PY  - 2026
DA  - 2026/10/27
JO  - Journal of the American Medical Informatics Association
DO  - 10.1093/jamia/ocag186
UR  - https://doi.org/10.1093/jamia/ocag186
AB  - A major concern in applying large language models (LLMs) to medicine is their reliability. Because LLMs generate text by sampling the next token (or word) from a probability distribution, the stochastic nature of this process can lead to different outputs even when the input prompt, model architecture, and parameters remain the same. Variation in model output has important implications for reliability in medical applications, yet it remains underexplored and lacks standardized metrics. To address this gap, we propose a statistical framework that systematically quantifies LLM variability using two metrics: repeatability, the consistency of LLM responses across repeated runs under identical conditions, and reproducibility, the consistency across runs under different conditions. Within these metrics, we evaluate two complementary dimensions: semantic consistency, which measures the similarity in meaning across responses, and internal stability, which measures the stability of the model's underlying token-generating process. We applied this framework to medical reasoning as a use case, evaluating LLM repeatability and reproducibility on standardized United States Medical Licensing Examination (USMLE) questions and real-world rare disease cases from the Undiagnosed Diseases Network (UDN) using validated medical reasoning prompts. LLM responses were less variable for UDN cases than for USMLE questions, suggesting that the complexity and ambiguity of real-world patient presentations may constrain the model's output space and yield more stable reasoning. Repeatability and reproducibility did not correlate with diagnostic accuracy, underscoring that an LLM producing a correct answer is not equivalent to producing it consistently. By providing a systematic approach to quantifying LLM repeatability and reproducibility, our framework supports more reliable use of LLMs in medicine and biomedical research.
SN  - 1067-5027
SN  - 1527-974X
ER  -