Scoring the NIH Stroke Scale from bedside exam transcripts: a synthetic benchmark of a decision model and a large language model
- 1Wayne State University
Abstract
BackgroundClinicians score the NIH Stroke Scale (NIHSS) at the bedside, often under time pressure, and ambient AI tools that listen to clinical encounters are now entering practice. It is not known whether software can score the NIHSS from the exam conversation itself, rather than extract a score that has already been written in a note.
MethodsWe created 112 synthetic code-stroke cases. One hundred were allocated in proportion to real code-stroke presentations, and 12 stress cases tested special scoring rules. For each case, a large language model (Claude Opus 5.5) wrote a reference score form (the answer key) and a speech-to-text-style exam transcript, with main-set transcripts split evenly between a narrated and a telegraphic examiner style. A second model (Claude Fable 5.1), blind to the key, scored every transcript and agreed with the key on 99.2% of items. Each transcript was then scored by Jev, a decision model, and by gpt-6-luna, a general large language model (LLM). The primary outcome was item-level accuracy against the key.
ResultsThe LLM scored 99.0% of items correctly (95% CI 98.4-99.4) and Jev 94.4% (93.2-95.4; McNemar p < 0.001), with exact totals in 101 and 63 of 112 cases. Jev was faster (median 0.15 vs 5.78 s per exam) and cheaper (USD 0.23 vs 0.67 per 1,000 exams). With telegraphic examiners, accuracy fell from 98.0% to 90.9% for Jev and from 100% to 97.7% for the LLM. In a post-hoc analysis that kept only the examiners words, accuracy stayed high when the examiner narrated (Jev 96.9%, LLM 98.8%) but dropped to about 80% when the examiner used shorthand. Most of Jevs errors fell on four items (language, extinction and neglect, LOC questions and visual fields), and on each it tended to repeat the same mistake, for example scoring slurred speech as aphasia. Its confidence flagged most of these errors. Re-checking the 14.8% of items with confidence below 0.70 would have caught 86% of them, and a simulated hybrid reached 98.8% accuracy.
ConclusionsBoth engines scored most NIHSS items correctly from exam transcripts. The LLM was more accurate, and the decision model was faster and cheaper, with a confidence signal that could route uncertain items to an LLM. Because the cases and the key are synthetic, these findings need confirmation with recorded exams and human raters.
Related
- Privacy-Aware Distillation of Large Language Models for Enhanced Multimorbidity Scoring
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- Evaluating a Large Language Model for Emergency Triage Using Synthetic Clinical Dialogues: Toward a Multimodal Clinical Safety Framework
- PsyEval: a comprehensive large language model evaluation benchmark for mental health
BibTeX
@misc{abood2026scoring,
title = {{Scoring the NIH Stroke Scale from bedside exam transcripts: a synthetic benchmark of a decision model and a large language model}},
author = {W. Abood and N. Abdulla},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.23.26366376},
url = {https://www.medrxiv.org/content/10.64898/2026.10.23.26366376v1},
note = {CC BY-ND},
}
RIS
TY - UNPB
TI - Scoring the NIH Stroke Scale from bedside exam transcripts: a synthetic benchmark of a decision model and a large language model
AU - W. Abood
AU - N. Abdulla
PY - 2026
DA - 2026/10/23
PB - medRxiv
DO - 10.64898/2026.10.23.26366376
UR - https://www.medrxiv.org/content/10.64898/2026.10.23.26366376v1
AB - BackgroundClinicians score the NIH Stroke Scale (NIHSS) at the bedside, often under time pressure, and ambient AI tools that listen to clinical encounters are now entering practice. It is not known whether software can score the NIHSS from the exam conversation itself, rather than extract a score that has already been written in a note. MethodsWe created 112 synthetic code-stroke cases. One hundred were allocated in proportion to real code-stroke presentations, and 12 stress cases tested special scoring rules. For each case, a large language model (Claude Opus 5.5) wrote a reference score form (the answer key) and a speech-to-text-style exam transcript, with main-set transcripts split evenly between a narrated and a telegraphic examiner style. A second model (Claude Fable 5.1), blind to the key, scored every transcript and agreed with the key on 99.2% of items. Each transcript was then scored by Jev, a decision model, and by gpt-6-luna, a general large language model (LLM). The primary outcome was item-level accuracy against the key. ResultsThe LLM scored 99.0% of items correctly (95% CI 98.4-99.4) and Jev 94.4% (93.2-95.4; McNemar p < 0.001), with exact totals in 101 and 63 of 112 cases. Jev was faster (median 0.15 vs 5.78 s per exam) and cheaper (USD 0.23 vs 0.67 per 1,000 exams). With telegraphic examiners, accuracy fell from 98.0% to 90.9% for Jev and from 100% to 97.7% for the LLM. In a post-hoc analysis that kept only the examiners words, accuracy stayed high when the examiner narrated (Jev 96.9%, LLM 98.8%) but dropped to about 80% when the examiner used shorthand. Most of Jevs errors fell on four items (language, extinction and neglect, LOC questions and visual fields), and on each it tended to repeat the same mistake, for example scoring slurred speech as aphasia. Its confidence flagged most of these errors. Re-checking the 14.8% of items with confidence below 0.70 would have caught 86% of them, and a simulated hybrid reached 98.8% accuracy. ConclusionsBoth engines scored most NIHSS items correctly from exam transcripts. The LLM was more accurate, and the decision model was faster and cheaper, with a confidence signal that could route uncertain items to an LLM. Because the cases and the key are synthetic, these findings need confirmation with recorded exams and human raters.
ER -