QoQ-Med3: a multimodal reasoning foundation model for clinical analysis.
10 affiliations
- 1MIT Media Lab, MIT, Cambridge, MA, USA.
- 2Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA.
- 3Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA.
- 4Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
- 5Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA.
- 6School of Medicine, Johns Hopkins University, Baltimore, MD, USA.
- 7Institute for Medical Engineering & Science, MIT, Cambridge, MA, USA.
- 8Martinos Center for Biomedical Imaging, Harvard Medical School, Boston, MA, USA.
- 9MIT Media Lab, MIT, Cambridge, MA, USA. [email protected].
- 10Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA. [email protected].
Abstract
Multimodal reasoning-based foundation models (MRFMs) hold considerable promise for addressing key challenges in medical practice, yet their readiness for real-world deployment remains insufficiently explored. To bridge this gap, we developed two MRFMs (QoQ-Med3 and QoQ-Med3-MIMIC) and systematically evaluated their (i) generalizability to previously unseen clinical modalities and tasks, (ii) transferability to held-out datasets collected across different clinical sites, and (iii) robustness to real-world challenges like cross-site heterogeneity. Our results demonstrate that these models can learn transferable representations across modalities, tasks, and heterogeneous clinical datasets. QoQ-Med3 achieves an overall balanced accuracy of 71.3% and a F1 of 0.349, superceding all open-source and closed-source models, including GPT-4o, with particularly pronounced gains in understudied modalities such as ultrasound and mammography. The model trained on public clinical data only generalized to both the held-out MIMIC-IV and the private JHU PMAP dataset collected at Johns Hopkins University hospital. In addition, the extrinsic hallucination rates are reduced by 44.4 percent after training. Collectively, our findings highlight both the potential of multimodal reasoning-based clinical foundation models and the critical next steps required to make them robust and reliable for real-world deployment.
Related
- Advances in Multimodal Artificial Intelligence in Radiology: Data Integration, Foundation Models, and Clinical Applications-A Narrative Review.
- From task-specific AI to multimodal foundation models in PET/CT: a readiness framework for clinical translation.
- Temporal-spatial reasoning over hypergraph knowledge structures for multimodal retrieval-augmented generation
- Artificial Intelligence-Driven Nanobody Engineering for Pathogen Recognition: From Functional Epitope Prioritization to Multimodal Foundation Models
- Cancer Informatics 2025: Bringing foundation models and multimodal cancer data to the forefront
BibTeX
@article{dai2026qoqmed3,
title = {{QoQ-Med3: a multimodal reasoning foundation model for clinical analysis.}},
author = {David Dai and Jeannie She and Jiaee Cheong and Xing Han and Carl Harris and Haowen Wei and Farzan Vahedifard and Suchi Saria and Robert Stevens and Paul Liang},
journal = {NPJ digital medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-02945-3},
url = {https://europepmc.org/article/MED/42926230},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - QoQ-Med3: a multimodal reasoning foundation model for clinical analysis.
AU - David Dai
AU - Jeannie She
AU - Jiaee Cheong
AU - Xing Han
AU - Carl Harris
AU - Haowen Wei
AU - Farzan Vahedifard
AU - Suchi Saria
AU - Robert Stevens
AU - Paul Liang
PY - 2026
DA - 2026/07/01
JO - NPJ digital medicine
DO - 10.1038/s41746-026-02945-3
UR - https://europepmc.org/article/MED/42926230
AB - Multimodal reasoning-based foundation models (MRFMs) hold considerable promise for addressing key challenges in medical practice, yet their readiness for real-world deployment remains insufficiently explored. To bridge this gap, we developed two MRFMs (QoQ-Med3 and QoQ-Med3-MIMIC) and systematically evaluated their (i) generalizability to previously unseen clinical modalities and tasks, (ii) transferability to held-out datasets collected across different clinical sites, and (iii) robustness to real-world challenges like cross-site heterogeneity. Our results demonstrate that these models can learn transferable representations across modalities, tasks, and heterogeneous clinical datasets. QoQ-Med3 achieves an overall balanced accuracy of 71.3% and a F1 of 0.349, superceding all open-source and closed-source models, including GPT-4o, with particularly pronounced gains in understudied modalities such as ultrasound and mammography. The model trained on public clinical data only generalized to both the held-out MIMIC-IV and the private JHU PMAP dataset collected at Johns Hopkins University hospital. In addition, the extrinsic hallucination rates are reduced by 44.4 percent after training. Collectively, our findings highlight both the potential of multimodal reasoning-based clinical foundation models and the critical next steps required to make them robust and reliable for real-world deployment.
SN - 2398-6352
ER -