Toward trustworthy diagnostic AI: debiasing and grounding large language models for seizure semiology interpretation
- 1Unit 1, Department of Neurology, Longhua Hospital, Shanghai University of Traditional Chinese Medicine
- 2Shanghai Minhang District Traditional Chinese Medicine Hospital
Abstract
Large language models (LLMs) have been proposed to map free-text seizure semiology to seizure-onset-zone (SOZ) likelihoods, but it remains unclear whether inference-time interventions can improve on the original SemioLLM benchmark without additional training. We evaluated two such interventions, candidate-order permutation within self-consistency and TF-IDF retrieval of case-specific labelled examples, individually and in combination (retrieval-augmented order-invariant self-consistency, RAOI-SC), across seven locally served open-weight LLMs on 1,268 Semio2Brain rows. Performance was assessed with weighted- and macro-F1, paired bootstrap confidence intervals, and exact McNemar tests against the self-consistency baseline, and a multi-dimensional reliability profile covering citation coverage and conditional verifiability, retrieval agreement, strict schema adherence, and confidence concentration; a TF-IDF nearest-neighbor classifier and grouped-split sensitivity analyses bounded how much of any gain could reflect retrieval leakage rather than the intervention itself. RAOI-SC had a higher weighted-F1 point estimate than self-consistency for six of seven models, but the improvement was supported by paired confidence intervals for four models and by McNemar testing for only one, Mistral-7B-Instruct-v0.3; the TF-IDF nearest-neighbor baseline alone achieved 55.72 weighted-F1 under row-level splitting, falling to 50.43 and 48.97 once identical-text and publication-reference duplicates were grouped together. These results support a cautious, model-specific reading of retrieval and order permutation as testable prompting interventions rather than a universally superior or clinically validated method, and define a row-level benchmark that does not by itself establish patient- or source-level generalization.
Related
- Effects of a Web-Based Debiasing Intervention on Brazilian Psychologists’ Accuracy on Major Depressive Disorder Diagnostic Tasks: A Randomized Training Intervention
- Design and Evaluation of a Source-Grounded Medical LLM for Clinical Decision Support and Patient Care in Trustworthy Diagnostic Systems.
- CT-Referenced Diagnostic Performance and Complementary Error Patterns of Artificial Intelligence Versus a Single Emergency Physician in Chest Radiograph Interpretation: A Retrospective Paired Diagnostic Accuracy Study
- Diagnostic accuracy of respiratory polygraphy interpretation for pediatric obstructive sleep apnea using a built-in algorithm
- Reducing Diagnostic Variation With Hybrid AI Coronary CTA Interpretation
BibTeX
@article{yuan2026toward,
title = {{Toward trustworthy diagnostic AI: debiasing and grounding large language models for seizure semiology interpretation}},
author = {Xiaolei Yuan and Xinyue Zhang and Hui Xu and Qing Ye and Canxing Yuan and Hui Li},
journal = {Frontiers in Neurology},
year = {2026},
month = oct,
doi = {10.3389/fneur.2026.1990185},
url = {https://www.frontiersin.org/articles/10.3389/fneur.2026.1990185/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Toward trustworthy diagnostic AI: debiasing and grounding large language models for seizure semiology interpretation
AU - Xiaolei Yuan
AU - Xinyue Zhang
AU - Hui Xu
AU - Qing Ye
AU - Canxing Yuan
AU - Hui Li
PY - 2026
DA - 2026/10/29
JO - Frontiers in Neurology
DO - 10.3389/fneur.2026.1990185
UR - https://www.frontiersin.org/articles/10.3389/fneur.2026.1990185/full
AB - Large language models (LLMs) have been proposed to map free-text seizure semiology to seizure-onset-zone (SOZ) likelihoods, but it remains unclear whether inference-time interventions can improve on the original SemioLLM benchmark without additional training. We evaluated two such interventions, candidate-order permutation within self-consistency and TF-IDF retrieval of case-specific labelled examples, individually and in combination (retrieval-augmented order-invariant self-consistency, RAOI-SC), across seven locally served open-weight LLMs on 1,268 Semio2Brain rows. Performance was assessed with weighted- and macro-F1, paired bootstrap confidence intervals, and exact McNemar tests against the self-consistency baseline, and a multi-dimensional reliability profile covering citation coverage and conditional verifiability, retrieval agreement, strict schema adherence, and confidence concentration; a TF-IDF nearest-neighbor classifier and grouped-split sensitivity analyses bounded how much of any gain could reflect retrieval leakage rather than the intervention itself. RAOI-SC had a higher weighted-F1 point estimate than self-consistency for six of seven models, but the improvement was supported by paired confidence intervals for four models and by McNemar testing for only one, Mistral-7B-Instruct-v0.3; the TF-IDF nearest-neighbor baseline alone achieved 55.72 weighted-F1 under row-level splitting, falling to 50.43 and 48.97 once identical-text and publication-reference duplicates were grouped together. These results support a cautious, model-specific reading of retrieval and order permutation as testable prompting interventions rather than a universally superior or clinically validated method, and define a row-level benchmark that does not by itself establish patient- or source-level generalization.
SN - 1664-2295
ER -