Modeling hierarchical response accuracy of large language models using a Bayesian generalized linear mixed model
- 1Faculty of Medicine, Department of Biostatistics, Izmir Katip Celebi University
- 2Faculty of Medicine, Department of Anatomy, Izmir Katip Celebi University
Abstract
Large Language Models (LLMs) demonstrate strong performance in Natural Language Processing. In neuroanatomy, their response accuracy across hierarchical data structures (responses nested within questions) has not yet been investigated. Evaluating how these models perform under varying cognitive demands is important for their effective integration into medical education. A total of 183 neuroanatomy questions from the Turkish Medical Specialization Examination (2013–2021) were reviewed, and 176 single correct answer questions were obtained by excluding seven figure-based questions. Each question was reformulated into prompts representing three role-based prompt levels: medical student, recently graduated physician, and doctoral-level specialist. Responses from the three LLMs were scored as correct or incorrect. Data were analyzed using a Bayesian Generalized Linear Mixed Model (GLMM) with a mixture prior for the random intercept, with model type, prompt level, and their interaction as fixed effects. The distribution of the question-level intercept was highly skewed, showing a ceiling for the items answered correctly by all LLMs under all prompt levels (141 of 176 questions, 80.1%). To deal with this excess of one, a spike and slab prior was specified for the random intercept, separating a ceiling component (spike) from a variable-difficulty component (slab). Posterior estimates were obtained using Monte Carlo sampling with CmdStan in Python. ChatGPT-4o and Microsoft Copilot showed higher overall accuracy than Google Gemini, with ChatGPT-4o performing best across all prompt levels. The spike and slab model showed better out-of-sample predictive performance than the standard model specifications (ΔELPD-WAIC = 22.64). The posterior global spike probability ( φ =0.706, 95% HDI [0.516, 0.826]) indicated that 141 questions (80.1%) were assigned to the ceiling component, while the remaining 35 questions (19.9%) showed true variability in item difficulty. The question level random intercept standard deviation for the slab component ( σ question =2.158, 95% HDI [1.289, 3.312]) reflected substantial between item variability in baseline easiness among non-ceiling neuroanatomical questions. The Bayesian GLMM approach implemented with a spike and slab prior distribution modeled the hierarchical structure of the data where responses were nested within questions. The ceiling effect was addressed by decomposing question level random intercepts into separate components representing the ceiling and variable difficulty providing an estimate of performance variability. It was obtained between-item variability in LLM performance among non-ceiling questions. Role-based prompt differentiation alone was insufficient to generate systematic differences in model accuracy.
Related
- How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Accuracy Overstates Evidence Grounding and Abstention Reliability in Mammography Vision-Language Models
- Nutritional Accuracy and Clinical Applicability of Large Language Model-Generated Therapeutic Diet Plans for Phenylketonuria: A Comparative Simulation Study
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
BibTeX
@article{emir2026modeling,
title = {{Modeling hierarchical response accuracy of large language models using a Bayesian generalized linear mixed model}},
author = {Busra Emir and Gonca Ay Keselik},
journal = {Scientific Reports},
year = {2026},
month = sep,
doi = {10.1038/s41598-026-71938-9},
url = {https://doi.org/10.1038/s41598-026-71938-9},
note = {CC BY},
}
RIS
TY - JOUR
TI - Modeling hierarchical response accuracy of large language models using a Bayesian generalized linear mixed model
AU - Busra Emir
AU - Gonca Ay Keselik
PY - 2026
DA - 2026/09/01
JO - Scientific Reports
DO - 10.1038/s41598-026-71938-9
UR - https://doi.org/10.1038/s41598-026-71938-9
AB - Large Language Models (LLMs) demonstrate strong performance in Natural Language Processing. In neuroanatomy, their response accuracy across hierarchical data structures (responses nested within questions) has not yet been investigated. Evaluating how these models perform under varying cognitive demands is important for their effective integration into medical education. A total of 183 neuroanatomy questions from the Turkish Medical Specialization Examination (2013–2021) were reviewed, and 176 single correct answer questions were obtained by excluding seven figure-based questions. Each question was reformulated into prompts representing three role-based prompt levels: medical student, recently graduated physician, and doctoral-level specialist. Responses from the three LLMs were scored as correct or incorrect. Data were analyzed using a Bayesian Generalized Linear Mixed Model (GLMM) with a mixture prior for the random intercept, with model type, prompt level, and their interaction as fixed effects. The distribution of the question-level intercept was highly skewed, showing a ceiling for the items answered correctly by all LLMs under all prompt levels (141 of 176 questions, 80.1%). To deal with this excess of one, a spike and slab prior was specified for the random intercept, separating a ceiling component (spike) from a variable-difficulty component (slab). Posterior estimates were obtained using Monte Carlo sampling with CmdStan in Python. ChatGPT-4o and Microsoft Copilot showed higher overall accuracy than Google Gemini, with ChatGPT-4o performing best across all prompt levels. The spike and slab model showed better out-of-sample predictive performance than the standard model specifications (ΔELPD-WAIC = 22.64). The posterior global spike probability ( φ =0.706, 95% HDI [0.516, 0.826]) indicated that 141 questions (80.1%) were assigned to the ceiling component, while the remaining 35 questions (19.9%) showed true variability in item difficulty. The question level random intercept standard deviation for the slab component ( σ _question =2.158, 95% HDI [1.289, 3.312]) reflected substantial between item variability in baseline easiness among non-ceiling neuroanatomical questions. The Bayesian GLMM approach implemented with a spike and slab prior distribution modeled the hierarchical structure of the data where responses were nested within questions. The ceiling effect was addressed by decomposing question level random intercepts into separate components representing the ceiling and variable difficulty providing an estimate of performance variability. It was obtained between-item variability in LLM performance among non-ceiling questions. Role-based prompt differentiation alone was insufficient to generate systematic differences in model accuracy.
SN - 2045-2322
ER -