Seeing through experts' eyes: a foundational vision-language model trained on radiologists' gaze and reasoning
6 affiliations
- 1Bioengineering Department and Imperial-X, Imperial College London, London, UK
- 2College of physics and information engineering, Fuzhou University, Fuzhou, China
- 3Department of Surgery and Cancer, Imperial College London, London, UK
- 4National Heart and Lung Institute, Imperial College London, London, UK
- 5Cardiovascular Research Centre, Royal Brompton Hospital, London, UK
- 6School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK
Abstract
Large-scale vision-language models have shown promise in automating chest X-ray interpretation. However, their clinical utility remains limited, since most systems optimize for semantic information rather than emulating how experts visually examine and interpret medical images. As a result, current models often overlook critical findings, misrepresent anatomical context, or diverge from established diagnostic workflows. Radiologists, by contrast, follow structured protocols that sequentially assess anatomical regions, reducing missed findings and supporting reliable diagnostic reasoning. We therefore introduce Gaze-X, a vision-language model that leverages radiologists’ eye-tracking data as a behavioral prior for expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, Gaze-X learns to follow the spatial and temporal structure of radiologist attention. Using a curated dataset of over 30,000 key frames from five radiologists interpreting chest X-rays across diverse disease categories, we show that Gaze-X produces more accurate, interpretable, and expert-consistent outputs across a range of clinically relevant tasks. Unlike autonomous reporting systems, Gaze-X produces verifiable evidence artifacts, including inspection trajectories and finding-linked localized regions, enabling transparent and safe human-AI collaboration. This capability provides a practical route toward more trustworthy, explainable, and diagnostically robust AI for radiology and beyond.
Related
- Zero-Shot Extraction of Structured Clinical Data from Free-Text CT Reports Using Foundational Large Language Models.
- Influence of large language models on pediatricians’ clinical reasoning and decision-making process
- Adjudicator sensitivity of safety estimates in large language models for open-ended cardiovascular reasoning: a blinded multi-adjudicator benchmark study running title: adjudicator variability in LLM safety
- No significant differences between large language model prompting strategies and human experts in summarisation of 'the formal EU-US meniscus rehabilitation consensus: An ESSKA-AOSSM-AASPT Initiative'.
- Enhancing Evidence Translation Through Structured Clinical Reasoning: A Mixed-Methods Controlled Feasibility Study of Peer Visibility and Readiness Mechanisms in Applied Learning
BibTeX
@article{lee2026seeing,
title = {{Seeing through experts' eyes: a foundational vision-language model trained on radiologists' gaze and reasoning}},
author = {Kinhei Lee and Peiyuan Jing and Zhenxuan Zhang and Yue Yang and Tao Wang and Dominic C. Marshall and Yingying Fang and Guang Yang},
journal = {Npj Artificial Intelligence},
year = {2026},
month = jul,
doi = {10.1038/s44387-026-00136-9},
url = {https://doi.org/10.1038/s44387-026-00136-9},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Seeing through experts' eyes: a foundational vision-language model trained on radiologists' gaze and reasoning
AU - Kinhei Lee
AU - Peiyuan Jing
AU - Zhenxuan Zhang
AU - Yue Yang
AU - Tao Wang
AU - Dominic C. Marshall
AU - Yingying Fang
AU - Guang Yang
PY - 2026
DA - 2026/07/13
JO - Npj Artificial Intelligence
DO - 10.1038/s44387-026-00136-9
UR - https://doi.org/10.1038/s44387-026-00136-9
AB - Large-scale vision-language models have shown promise in automating chest X-ray interpretation. However, their clinical utility remains limited, since most systems optimize for semantic information rather than emulating how experts visually examine and interpret medical images. As a result, current models often overlook critical findings, misrepresent anatomical context, or diverge from established diagnostic workflows. Radiologists, by contrast, follow structured protocols that sequentially assess anatomical regions, reducing missed findings and supporting reliable diagnostic reasoning. We therefore introduce Gaze-X, a vision-language model that leverages radiologists’ eye-tracking data as a behavioral prior for expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, Gaze-X learns to follow the spatial and temporal structure of radiologist attention. Using a curated dataset of over 30,000 key frames from five radiologists interpreting chest X-rays across diverse disease categories, we show that Gaze-X produces more accurate, interpretable, and expert-consistent outputs across a range of clinically relevant tasks. Unlike autonomous reporting systems, Gaze-X produces verifiable evidence artifacts, including inspection trajectories and finding-linked localized regions, enabling transparent and safe human-AI collaboration. This capability provides a practical route toward more trustworthy, explainable, and diagnostically robust AI for radiology and beyond.
SN - 3005-1460
ER -