{
  "abstract": "While multimodal large language models (LLMs) demonstrate significant potential in healthcare applications, their clinical utility is difficult to appraise. Current evaluations of medical-assisting LLMs are often limited by sparse human expertise, narrow specialty scope, and reliance on multiple-choice benchmarks or synthetic vignettes, which can inflate performance and obscure clinical utility. We conducted a multicenter, multidisciplinary study in which more than 400 physicians—spanning seven specialties, varied experience levels, and multiple geographic settings—evaluated LLM-generated free-text responses to real, de-identified clinical cases. In a matched-control design, we also deployed an equivalent number of AI agents configured to mirror physician characteristics to examine whether automated evaluators can supplement or replace human assessment. Our results demonstrated that physician assessments exhibited substantial heterogeneity by clinical seniority and practice environment, leading to notable shifts in relative model rankings across cohorts. While AI agents delivered highly efficient, directionally aligned assessments, they did not fully capture the nuances of human clinical judgment and could not substitute for physician-centered evaluation. Instead, they promise assistive tools that can triage or pre-screen outputs to reduce human burden.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Biomedical Engineering, The Chinese University of Hong Kong",
    "Department of Interventional Radiology, Beijing Friendship Hospital, Capital Medical University",
    "Department of Metabolism, Digestion and Reproduction, Imperial College London",
    "Department of Radiology, Qinghai University Affiliated Hospital, Qinghai University",
    "Institute of Ophthalmology, University College London",
    "Department of Hepatobiliary Surgery, Xijing Hospital, Fourth Military Medical University",
    "State Key Laboratory of Pathogenesis, Prevention and Treatment of High Incidence Diseases in Central Asia, Clinical Medicine Institute, The First Affiliated Hospital of Xinjiang Medical University",
    "Department of Ophthalmology, Beijing Friendship Hospital, Capital Medical University",
    "Department of Anthropology, The Chinese University of Hong Kong",
    "Artificial Intelligence Thrust, The Hong Kong University of Science and Technology (HKUST)",
    "The Hamlyn Centre for Robotic Surgery, Imperial College London",
    "Department of Cardiology, The 7th People’s Hospital of Zhengzhou",
    "Department of Cardiology, The First Affiliated Hospital, Sun-yat Sen University",
    "Department of Cardiology, China-Japan Union Hospital of Jilin University",
    "Department of Hepatobiliary and Pancreatic Surgery, Qinghai University Affiliated Hospital",
    "Department of Radiology, Lanzhou University Second Hospital",
    "Department of Cardiology, Zhongshan Hospital, Fudan University",
    "Department of Medical Imaging, Henan Provincial People’s Hospital",
    "Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University"
  ],
  "article_type": null,
  "arxiv": null,
  "author_total": 28,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Peilun Shi"
    },
    {
      "affs": [
        1
      ],
      "name": "Jian Li"
    },
    {
      "affs": [
        2
      ],
      "name": "Ziqi Yang"
    },
    {
      "affs": [
        3
      ],
      "name": "Yuntai Cao"
    },
    {
      "affs": [
        4
      ],
      "name": "Lie Ju"
    },
    {
      "affs": [
        3
      ],
      "name": "Haihua Bao"
    },
    {
      "affs": [
        5
      ],
      "name": "Hao Xu"
    },
    {
      "affs": [
        6
      ],
      "name": "Yichen Fan"
    },
    {
      "affs": [
        7
      ],
      "name": "Tianmin Ren"
    },
    {
      "affs": [
        8
      ],
      "name": "Yiyang Xiao"
    },
    {
      "affs": [
        9
      ],
      "name": "Jinjing Zhu"
    },
    {
      "affs": [
        10
      ],
      "name": "Junhong Chen"
    },
    {
      "affs": [
        11
      ],
      "name": "Yong Dong"
    },
    {
      "affs": [
        12
      ],
      "name": "Xun Hu"
    },
    {
      "affs": [
        13
      ],
      "name": "Yaliang Tong"
    },
    {
      "affs": [
        14
      ],
      "name": "Zhan Wang"
    },
    {
      "affs": [
        2
      ],
      "name": "Yuchen Xiang"
    },
    {
      "affs": [
        2
      ],
      "name": "Jinshi Zhao"
    },
    {
      "affs": [
        15
      ],
      "name": "Junlin Zhou"
    },
    {
      "affs": [
        2
      ],
      "name": "Burak Temelkuran"
    },
    {
      "affs": [
        4
      ],
      "name": "Yukun Zhou"
    },
    {
      "affs": [
        1
      ],
      "name": "Long Jin"
    },
    {
      "affs": [
        16
      ],
      "name": "Chenguang Li"
    },
    {
      "affs": [
        17
      ],
      "name": "Meiyun Wang"
    },
    {
      "affs": [
        9
      ],
      "name": "Hui Xiong"
    },
    {
      "affs": [
        4
      ],
      "name": "Pearse A. Keane"
    },
    {
      "affs": [
        16
      ],
      "name": "Junbo Ge"
    },
    {
      "affs": [
        18
      ],
      "name": "Ningli Wang"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/dpt3tcg2jc70z.bib",
  "cited_by": 2,
  "doi": "10.1038/s41746-026-02942-6",
  "fulltext_url": "https://doi.org/10.1038/s41746-026-02942-6",
  "id": "dpt3tcg2jc70z",
  "issns": [
    "2398-6352"
  ],
  "landing_url": "https://doi.org/10.1038/s41746-026-02942-6",
  "language": "en",
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "lcc",
      "R858-859.7"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7167083232",
  "pdf_url": "https://www.nature.com/articles/s41746-026-02942-6_reference.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-07-01",
  "publisher": "Nature Portfolio",
  "ris": "https://research.draconitic.com/paper/dpt3tcg2jc70z.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T13:46:20Z",
      "native_id": "3ce46be4bc6f49f591fd769ff6026c31",
      "source": "doaj",
      "url": "https://doi.org/10.1038/s41746-026-02942-6"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "openalex:2718|Artificial Intelligence in Healthcare and Education",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases",
  "title_text": "Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T13:46:20Z",
  "url": "https://research.draconitic.com/paper/dpt3tcg2jc70z",
  "venue": "npj Digital Medicine",
  "versions": []
}