Entity-centric evaluation of large language model responses for medical question-answering tasks
- 1Faculty of Computing & Data Sciences, Boston University, Boston, Massachusetts, United States of America.
- 2Department of Medicine, Boston University Chobanian & Avedisian School of Medicine, Boston, Massachusetts, United States of America.
- 3Department of Computer Science, Boston University, Boston, Massachusetts, United States of America.
Abstract
Large language models (LLMs) are increasingly evaluated for clinical question-answering (QA) tasks, yet benchmark accuracy alone provides limited assurance of clinical reliability. Existing evaluation metrics often fail to capture whether model reasoning preserves patient-specific context and diagnostic intent, while many require external references or manual annotation, limiting scalability and real-world applicability. To address this gap, this study proposes EntQA, a reference-free, entity-centric metric that evaluates how well LLM-generated responses retain clinically relevant biomedical concepts from patient backgrounds and diagnostic questions. Across five medical QA benchmarks and seven Qwen 2.5 Instruct models (0.5B–72B parameters), EntQA showed consistently positive associations with model accuracy and scaling, outperforming conventional overlap- and embedding-based metrics that frequently exhibited weak or negative correlations. Group-level correlations with accuracy reached Spearman ρ =0.9286, while correlations with model scale reached ρ = 0.252. These findings suggest that EntQA provides a scalable and interpretable framework for assessing clinical fidelity and reasoning quality in healthcare LLMs without requiring gold standard references or external evidence. Author summary: Large language models (LLMs) are starting to appear in hospitals and clinics, where they help doctors answer questions and review patient information. However, these tools can sometimes miss important details from a patient’s history or the specific question being asked, which could lead to incomplete or misleading advice. Most current ways of checking these models only look at whether the final answer matches a reference. They often miss cases where the model ignored key clinical information from the original case. To help solve this problem, we created EntQA, a new way to evaluate how well LLM responses keep and use the important medical details from both the patient’s background and the question itself. EntQA works by identifying key medical concepts and checking which ones appear in the LLM’s answer. When we tested it across several well-known medical question-answering benchmarks, EntQA gave a clearer picture of how good the LLM responses really were compared with traditional scoring methods. It also makes it easier to see exactly which clinical details the model got right or missed. We believe tools like EntQA can help researchers and developers build safer, more reliable AI systems for healthcare.
Related
- MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering
- Relation-aware dynamic knowledge subgraph enhanced large language models for medical multiple-choice question answering
- Version-aware question answering over living oncology guidelines
- LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
- Answering clinicians' questions over trial evidence tables with verifiable, feedback-driven language models
BibTeX
@article{liu2026entitycentric,
title = {{Entity-centric evaluation of large language model responses for medical question-answering tasks}},
author = {Yi Liu and Vijaya B Kolachalama},
journal = {PLOS Digital Health},
year = {2026},
month = oct,
doi = {10.1371/journal.pdig.0001752},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001752},
note = {CC BY},
}
RIS
TY - JOUR
TI - Entity-centric evaluation of large language model responses for medical question-answering tasks
AU - Yi Liu
AU - Vijaya B Kolachalama
PY - 2026
DA - 2026/10/06
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001752
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001752
AB - Large language models (LLMs) are increasingly evaluated for clinical question-answering (QA) tasks, yet benchmark accuracy alone provides limited assurance of clinical reliability. Existing evaluation metrics often fail to capture whether model reasoning preserves patient-specific context and diagnostic intent, while many require external references or manual annotation, limiting scalability and real-world applicability. To address this gap, this study proposes EntQA, a reference-free, entity-centric metric that evaluates how well LLM-generated responses retain clinically relevant biomedical concepts from patient backgrounds and diagnostic questions. Across five medical QA benchmarks and seven Qwen 2.5 Instruct models (0.5B–72B parameters), EntQA showed consistently positive associations with model accuracy and scaling, outperforming conventional overlap- and embedding-based metrics that frequently exhibited weak or negative correlations. Group-level correlations with accuracy reached Spearman ρ =0.9286, while correlations with model scale reached ρ = 0.252. These findings suggest that EntQA provides a scalable and interpretable framework for assessing clinical fidelity and reasoning quality in healthcare LLMs without requiring gold standard references or external evidence. Author summary: Large language models (LLMs) are starting to appear in hospitals and clinics, where they help doctors answer questions and review patient information. However, these tools can sometimes miss important details from a patient’s history or the specific question being asked, which could lead to incomplete or misleading advice. Most current ways of checking these models only look at whether the final answer matches a reference. They often miss cases where the model ignored key clinical information from the original case. To help solve this problem, we created EntQA, a new way to evaluate how well LLM responses keep and use the important medical details from both the patient’s background and the question itself. EntQA works by identifying key medical concepts and checking which ones appear in the LLM’s answer. When we tested it across several well-known medical question-answering benchmarks, EntQA gave a clearer picture of how good the LLM responses really were compared with traditional scoring methods. It also makes it easier to see exactly which clinical details the model got right or missed. We believe tools like EntQA can help researchers and developers build safer, more reliable AI systems for healthcare.
SN - 2767-3170
ER -