Skip to content
DraconiticResearch
Search Sources API
223,421 studies31 sourcesUpdated 41 min ago
Peer-reviewedCC BY-NC-ND 4.0Diagnostic Pathology

Reasoning beyond accuracy: expert evaluation of expressed diagnostic reasoning in large language model outputs for pathology

  • Asim Waqas1,2
  • Ehsan Ullah3
  • Asma Khan4
  • Farah Khalil5
  • Zarifa G. Ozturk6
  • Vaibhav Chumbalkar5
  • Daryoush Saeed-Vafa5
  • Zena Jameel5
  • Wei-Shen Chen7
  • Humberto Trejo Bittar5
  • Jasreman Dhillon8
  • Rajendra S. Singh9
5 more authors
  • Andrey Bychkov10
  • Anil V. Parwani11
  • Marilyn M. Bui5
  • Matthew B. Schabath1
  • Ghulam Rasool2
11 affiliations
  1. 1Department of Cancer Epidemiology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, 33612 FL USA
  2. 2Department of Machine Learning, H. Lee Moffitt Cancer Center & Research Institute, Tampa, 33612 FL USA
  3. 3Department of Surgery, Health New Zealand, Counties Manukau, Auckland, New Zealand
  4. 4Armed Forces Institute of Pathology, Rawalpindi, Pakistan
  5. 5Department of Pathology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, 33612 FL USA
  6. 6Clinical Science Lab, H. Lee Moffitt Cancer Center & Research Institute, Tampa, 33612 FL USA
  7. 7Department of Dermatology & Cutaneous Surgery, University of South Florida, Tampa, 33612 FL USA
  8. 8Department of Anatomic Pathology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, 33612 FL USA
  9. 9Dermatopathology and Digital Pathology, Summit Health, Berkley Heights, 07922 NJ USA
  10. 10Department of Pathology, Kameda Medical Center, Kamogawa City, Chiba Prefecture Japan
  11. 11Department of Pathology, The Ohio State University, Columbus, 43210 Ohio USA
Read full textDOICite

Abstract

Diagnostic pathology depends on complex, structured reasoning to interpret clinical, histologic, and molecular data. Replicating, or even approximating this cognitive process algorithmically remains a significant challenge. As large language models (LLMs) gain traction in medicine, it is critical to assess their clinical utility in specialized domains such as pathology. We evaluated the quality of expressed diagnostic reasoning in the outputs of four LLMs, OpenAI o1, OpenAI o3-mini, Gemini 2.0 Flash Thinking Experimental, and DeepSeek-R1, using 15 constructed-response pathology questions covering board-examination-level content across a range of cognitive demands, from factual recall to integrative knowledge synthesis. Eleven expert pathologists independently scored model-generated responses using a structured rubric encompassing five language quality metrics (accuracy, relevance, coherence, depth, and conciseness) and seven diagnostic reasoning strategies as communicated in the textual output. Scores were normalized and compared using ANOVA and Tukey’s HSD. Gemini significantly outperformed all models in overall reasoning quality, particularly in analytical depth and coherence. While all models achieved comparable factual accuracy, only Gemini and DeepSeek consistently demonstrated expert-like reasoning patterns in their outputs, including algorithmic, inductive, and Bayesian approaches. Performance varied by reasoning type, with the highest scores in algorithmic and deductive reasoning and the lowest in heuristic and pattern recognition. Gemini achieved the highest inter-observer agreement (p < 0.001), suggesting greater interpretability. However, more in-depth reasoning was generally associated with reduced conciseness. Addressing these trade-offs will be key to safe clinical integration.

Related

  1. Information theory and diagnostic reasoning in primary care: a framework for navigating uncertainty.
    Brian H Willis
    Family practicePeer-reviewedCC BYFull textGeneral medicineEurope PMC
  2. Stroke-Heart Syndrome: Mechanisms, Temporal Phenotypes, and Diagnostic Reasoning.
    Fa Lin, Jian Lyu, Yanci Liu +11
    Advanced science (Weinheim, Baden-Wurttemberg, Germany)Peer-reviewedCC BYFull textCardiovascular medicineEurope PMCPMC
  3. Clinical expertise and cognitive reflection predict fewer misconceptions in Bayesian diagnostic reasoning
    Markus H. Hefter, Nathan B. Hinkel, Andreas Hinkel
    BMC Medical EducationPeer-reviewedCC BY 4.0Full textGeneral medicineDOAJPMC
  4. Diffuse Alveolar Haemorrhage Masquerading as Worsening ARDS in a Young Polytrauma Patient: Diagnostic Reasoning and a Conservative Management Bundle - A Case Report.
    Zhaolve Huang, Minghao Yang, Jishen Luo +1
    Patient preference and adherencePeer-reviewedCC BY-NCFull textGeneral medicineEurope PMC
  5. Comparative evaluation of ChatGPT and Gemini feedback in scenario-based clinical reasoning among medical interns.
    Hilal Hatice Ülkü, Fulya Torun, Selcen Öncü
    Revista da Associacao Medica Brasileira (1992)Peer-reviewedCC BYFull textEpidemiology & public healthEurope PMC

Cite

BibTeXRIS
BibTeX
@article{waqas2026reasoning,
  title = {{Reasoning beyond accuracy: expert evaluation of expressed diagnostic reasoning in large language model outputs for pathology}},
  author = {Asim Waqas and Ehsan Ullah and Asma Khan and Farah Khalil and Zarifa G. Ozturk and Vaibhav Chumbalkar and Daryoush Saeed-Vafa and Zena Jameel and Wei-Shen Chen and Humberto Trejo Bittar and Jasreman Dhillon and Rajendra S. Singh and Andrey Bychkov and Anil V. Parwani and Marilyn M. Bui and Matthew B. Schabath and Ghulam Rasool},
  journal = {Diagnostic Pathology},
  year = {2026},
  month = jul,
  doi = {10.1186/s13000-026-01809-4},
  url = {https://doi.org/10.1186/s13000-026-01809-4},
  note = {CC BY-NC-ND 4.0},
}
RIS
TY  - JOUR
TI  - Reasoning beyond accuracy: expert evaluation of expressed diagnostic reasoning in large language model outputs for pathology
AU  - Asim Waqas
AU  - Ehsan Ullah
AU  - Asma Khan
AU  - Farah Khalil
AU  - Zarifa G. Ozturk
AU  - Vaibhav Chumbalkar
AU  - Daryoush Saeed-Vafa
AU  - Zena Jameel
AU  - Wei-Shen Chen
AU  - Humberto Trejo Bittar
AU  - Jasreman Dhillon
AU  - Rajendra S. Singh
AU  - Andrey Bychkov
AU  - Anil V. Parwani
AU  - Marilyn M. Bui
AU  - Matthew B. Schabath
AU  - Ghulam Rasool
PY  - 2026
DA  - 2026/07/29
JO  - Diagnostic Pathology
DO  - 10.1186/s13000-026-01809-4
UR  - https://doi.org/10.1186/s13000-026-01809-4
AB  - Diagnostic pathology depends on complex, structured reasoning to interpret clinical, histologic, and molecular data. Replicating, or even approximating this cognitive process algorithmically remains a significant challenge. As large language models (LLMs) gain traction in medicine, it is critical to assess their clinical utility in specialized domains such as pathology. We evaluated the quality of expressed diagnostic reasoning in the outputs of four LLMs, OpenAI o1, OpenAI o3-mini, Gemini 2.0 Flash Thinking Experimental, and DeepSeek-R1, using 15 constructed-response pathology questions covering board-examination-level content across a range of cognitive demands, from factual recall to integrative knowledge synthesis. Eleven expert pathologists independently scored model-generated responses using a structured rubric encompassing five language quality metrics (accuracy, relevance, coherence, depth, and conciseness) and seven diagnostic reasoning strategies as communicated in the textual output. Scores were normalized and compared using ANOVA and Tukey’s HSD. Gemini significantly outperformed all models in overall reasoning quality, particularly in analytical depth and coherence. While all models achieved comparable factual accuracy, only Gemini and DeepSeek consistently demonstrated expert-like reasoning patterns in their outputs, including algorithmic, inductive, and Bayesian approaches. Performance varied by reasoning type, with the highest scores in algorithmic and deductive reasoning and the lowest in heuristic and pattern recognition. Gemini achieved the highest inter-observer agreement (p < 0.001), suggesting greater interpretability. However, more in-depth reasoning was generally associated with reduced conciseness. Addressing these trade-offs will be key to safe clinical integration.
SN  - 1746-1596
ER  -