MedicalAgentsBench for complex medical reasoning: Comparing internalized reasoning models versus externalized agent-based frameworks
8 affiliations
- 1Department of Biomedical Informatics & Data Science, Yale University, New Haven, CT 06520, USA
- 2Program in Computational Biology & Biomedical Informatics, Yale University, New Haven, CT 06520, USA
- 3Department of Computer Science, Yale University, New Haven, CT 06520, USA
- 4Department of Biosystems Science and Engineering, ETH Zurich, 4058 Basel, Switzerland
- 5Department of Computer Science, Stanford University, Stanford, CA 94305, USA
- 6Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX 75390, USA
- 7Department of Molecular Biophysics and Biochemistry, Yale University, New Haven, CT 06520, USA
- 8Department of Statistics & Data Science, Yale University, New Haven, CT 06520, USA
Abstract
Complex medical reasoning requires integrating heterogeneous clinical evidence across multiple inference steps. Large language models (LLMs) now approach this through two routes: internalized reasoning and externalized agent scaffolding (frameworks that decompose problems collaboratively among multiple LLMs). To determine whether these routes are exclusive or complementary, we introduce MedicalAgentsBench, a filtered benchmark of 862 complex clinical questions drawn from the union of eight medical datasets via difficulty-aware curation and contamination screening. Evaluating three internalized reasoning models (DeepSeek-R1, o1-mini, and o3-mini), seven base models, and nine externalized agent-based methods, we find that internalized and externalized approaches each independently improve performance and that their benefits compound: the highest accuracy is achieved by layering agent workflows onto an internalized reasoning model (i.e., o3-mini + MDAgents with 35.1%). Pareto analysis shows this combination dominates the cost-performance frontier; moreover, lightweight optimization on inexpensive models offers an entry point for resource-constrained settings.
Related
- The Voice You Want, Not the Voice You Have: Chosen Singers as Externalized Idealized Selves
- Does internalized homonegativity lower sexual satisfaction? A longitudinal assessment among gay, bisexual and other men who have sex with men (GBM) in Canada's three largest cities
- Internalized weight stigma and compulsive exercise among college students: a variable-centered and person-centered analysis.
- Internalized Stigma and Its Associations with General and Family Functioning in Patients with Bipolar Disorder
- Intergenerational burden: deconstructing the dual impact of familial suicide in a specialized bipolar clinic, from clinical risk to internalized stigma
BibTeX
@article{shao2026medicalagentsbench,
title = {{MedicalAgentsBench for complex medical reasoning: Comparing internalized reasoning models versus externalized agent-based frameworks}},
author = {Yanjun Shao and Xiangru Tang and Jiwoong Sohn and Jiapeng Chen and Yuxuan Liao and Jiayi Zhang and J Xiang and Fang Wu and Yilun Zhao and Chenglin Wu and Wenqi Shi and Arman Cohan and Mark Gerstein},
journal = {Patterns (New York, N.Y.)},
year = {2026},
month = jul,
doi = {10.1016/j.patter.2026.101610},
url = {https://europepmc.org/article/PMC/PMC13674309},
note = {CC BY},
}
RIS
TY - JOUR
TI - MedicalAgentsBench for complex medical reasoning: Comparing internalized reasoning models versus externalized agent-based frameworks
AU - Yanjun Shao
AU - Xiangru Tang
AU - Jiwoong Sohn
AU - Jiapeng Chen
AU - Yuxuan Liao
AU - Jiayi Zhang
AU - J Xiang
AU - Fang Wu
AU - Yilun Zhao
AU - Chenglin Wu
AU - Wenqi Shi
AU - Arman Cohan
AU - Mark Gerstein
PY - 2026
DA - 2026/07/08
JO - Patterns (New York, N.Y.)
DO - 10.1016/j.patter.2026.101610
UR - https://europepmc.org/article/PMC/PMC13674309
AB - Complex medical reasoning requires integrating heterogeneous clinical evidence across multiple inference steps. Large language models (LLMs) now approach this through two routes: internalized reasoning and externalized agent scaffolding (frameworks that decompose problems collaboratively among multiple LLMs). To determine whether these routes are exclusive or complementary, we introduce MedicalAgentsBench, a filtered benchmark of 862 complex clinical questions drawn from the union of eight medical datasets via difficulty-aware curation and contamination screening. Evaluating three internalized reasoning models (DeepSeek-R1, o1-mini, and o3-mini), seven base models, and nine externalized agent-based methods, we find that internalized and externalized approaches each independently improve performance and that their benefits compound: the highest accuracy is achieved by layering agent workflows onto an internalized reasoning model (i.e., o3-mini + MDAgents with 35.1%). Pareto analysis shows this combination dominates the cost-performance frontier; moreover, lightweight optimization on inexpensive models offers an entry point for resource-constrained settings.
SN - 2666-3899
ER -