{ "abstract": "Large language models (LLMs) are increasingly evaluated for clinical question-answering (QA) tasks, yet benchmark accuracy alone provides limited assurance of clinical reliability. Existing evaluation metrics often fail to capture whether model reasoning preserves patient-specific context and diagnostic intent, while many require external references or manual annotation, limiting scalability and real-world applicability. To address this gap, this study proposes EntQA, a reference-free, entity-centric metric that evaluates how well LLM-generated responses retain clinically relevant biomedical concepts from patient backgrounds and diagnostic questions. Across five medical QA benchmarks and seven Qwen 2.5 Instruct models (0.5B–72B parameters), EntQA showed consistently positive associations with model accuracy and scaling, outperforming conventional overlap- and embedding-based metrics that frequently exhibited weak or negative correlations. Group-level correlations with accuracy reached Spearman ρ =0.9286, while correlations with model scale reached ρ = 0.252. These findings suggest that EntQA provides a scalable and interpretable framework for assessing clinical fidelity and reasoning quality in healthcare LLMs without requiring gold standard references or external evidence. Author summary: Large language models (LLMs) are starting to appear in hospitals and clinics, where they help doctors answer questions and review patient information. However, these tools can sometimes miss important details from a patient’s history or the specific question being asked, which could lead to incomplete or misleading advice. Most current ways of checking these models only look at whether the final answer matches a reference. They often miss cases where the model ignored key clinical information from the original case. To help solve this problem, we created EntQA, a new way to evaluate how well LLM responses keep and use the important medical details from both the patient’s background and the question itself. EntQA works by identifying key medical concepts and checking which ones appear in the LLM’s answer. When we tested it across several well-known medical question-answering benchmarks, EntQA gave a clearer picture of how good the LLM responses really were compared with traditional scoring methods. It also makes it easier to see exactly which clinical details the model got right or missed. We believe tools like EntQA can help researchers and developers build safer, more reliable AI systems for healthcare.", "abstract_shown": true, "affiliations": [ "Faculty of Computing & Data Sciences, Boston University, Boston, Massachusetts, United States of America.", "Department of Medicine, Boston University Chobanian & Avedisian School of Medicine, Boston, Massachusetts, United States of America.", "Department of Computer Science, Boston University, Boston, Massachusetts, United States of America." ], "article_type": "Research Article", "arxiv": null, "author_total": 2, "authors": [ { "affs": [ 0 ], "name": "Yi Liu" }, { "affs": [ 0, 1, 2 ], "name": "Vijaya B Kolachalama", "orcid": "0000-0002-5312-8644" } ], "bibtex": "https://research.draconitic.com/paper/0cb8y19z65707.bib", "cited_by": 0, "doi": "10.1371/journal.pdig.0001752", "fulltext_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001752", "id": "0cb8y19z65707", "issns": [ "2767-3170" ], "landing_url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001752", "language": "eng", "licence": "CC-BY", "licence_label": "CC BY", "licence_source": "article", "licence_url": "https://creativecommons.org/licenses/by/4.0/", "metadata_licence": null, "native_subjects": [ [ "openalex", "2718|Artificial Intelligence in Healthcare and Education" ], [ "plos", "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Language" ], [ "plos", "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Perception/Sensory perception/Hallucinations" ], [ "plos", "/Biology and life sciences/Neuroscience/Cognitive science/Cognitive psychology/Reasoning" ], [ "plos", "/Biology and life sciences/Neuroscience/Sensory perception/Hallucinations" ], [ "plos", "/Biology and life sciences/Psychology/Cognitive psychology/Language" ], [ "plos", "/Biology and life sciences/Psychology/Cognitive psychology/Perception/Sensory perception/Hallucinations" ], [ "plos", "/Biology and life sciences/Psychology/Cognitive psychology/Reasoning" ], [ "plos", "/Computer and information sciences/Information technology/Natural language processing/Named entity recognition" ], [ "plos", "/Medicine and health sciences" ], [ "plos", "/Medicine and health sciences/Diagnostic medicine" ], [ "plos", "/Medicine and health sciences/Health care/Health care providers/Physicians" ], [ "plos", "/People and places/Population groupings/Professions/Medical personnel/Physicians" ], [ "plos", "/Research and analysis methods/Database and informatics methods/Bioinformatics/Sequence analysis/Sequence alignment" ], [ "plos", "/Social sciences/Psychology/Cognitive psychology/Language" ], [ "plos", "/Social sciences/Psychology/Cognitive psychology/Perception/Sensory perception/Hallucinations" ], [ "plos", "/Social sciences/Psychology/Cognitive psychology/Reasoning" ] ], "openalex": "W7220658930", "pdf_url": "https://journals.plos.org/digitalhealth/article/file?id=10.1371/journal.pdig.0001752&type=printable", "pmcid": "PMC13641718", "pmid": "42837409", "published": "2026-10-06", "publisher": "Public Library of Science", "ris": "https://research.draconitic.com/paper/0cb8y19z65707.ris", "sources": [ { "first_seen": "2026-10-30T12:17:30Z", "native_id": "10.1371/journal.pdig.0001752", "source": "plos", "url": "https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001752" }, { "first_seen": "2026-10-30T12:18:03Z", "native_id": "MED:42837409", "source": "europepmc", "url": "https://europepmc.org/article/MED/42837409" } ], "subjects": [ { "field": "biology", "origin": "plos:neuroscience", "subfield": "neuroscience" } ], "title": "Entity-centric evaluation of large language model responses for medical question-answering tasks", "title_text": "Entity-centric evaluation of large language model responses for medical question-answering tasks", "type": "peer-reviewed", "updated_at": "2026-10-30T12:18:03Z", "url": "https://research.draconitic.com/paper/0cb8y19z65707", "venue": "PLOS Digital Health", "versions": [] }