Triage Advice From General-Purpose Language Models in Health and Everyday Situations
- 1University of Florence
Abstract
Importance: One adult in 10 has consulted a chatbot for health information within 6 months, and nearly half of users do so to decide whether they need care. Referral regardless of need could add demand on stretched services. Objective: To compare referral to a professional by general-purpose language models in health and non-health situations against the level of contact that human raters judged necessary. Design, Setting, and Participants: Cross-sectional study of 4,346 responses generated in September 2026 by 53 general-purpose model configurations, released between March 2023 and September 2026 and accessed through APIs or run locally, to 82 investigator-written messages (41 health and 41 non-health). No patients were involved. Exposures: Health vs non-health scenario. Main Outcomes and Measures: Professional referral, unconditional or conditional on a future change or on doubt, coded by 3 language models and compared with the level of contact that physicians, pharmacists and 2 technically trained raters judged necessary. Results: Referral occurred in 74.7% of health and 46.3% of non-health responses, more often for health in every configuration (standardized difference, 28.3 percentage points; 95% CI, 18.3-37.0). Where raters judged contact unnecessary, 64.5% of health vs 27.6% of non-health responses recommended a professional (odds ratio, 17.08; 95% CI, 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%), and unconditional referral was 18.1% vs 6.6% (odds ratio, 4.66; 95% CI, 1.43-15.19). Where an in-person assessment was judged necessary, referral was similar (89.6% vs 84.5%). In medicines and physical health, referral was not associated with rated severity (odds ratio per point, 1.16; 95% CI, 0.53-2.55), unlike outside health (3.86; 95% CI, 1.79-8.33). The referral gap was 20.5 percentage points among configurations released in 2023 and 32.2 and 32.3 in 2025 and 2026 (4.8 percentage points per year; 95% CI, 2.3-7.2). Conclusions and Relevance: General-purpose language models referred people to a professional where a visit was needed, in health as outside it, but in health also in 2 in 3 situations that physicians and pharmacists judged manageable without contact, a pattern stronger in newer models. Evaluations of these systems should include whether their advice distinguishes situations that can be deferred or managed at a distance, alongside the referral of serious cases.
Related
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- General-purpose large language models outperform specialized clinical AI tools on medical benchmarks
- Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
- Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
BibTeX
@misc{vannacci2026triage,
title = {{Triage Advice From General-Purpose Language Models in Health and Everyday Situations}},
author = {A. Vannacci and G. Crescioli and C. Ravaldi and R. Bonaiuti},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.31.26366978},
url = {https://www.medrxiv.org/content/10.64898/2026.10.31.26366978v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Triage Advice From General-Purpose Language Models in Health and Everyday Situations
AU - A. Vannacci
AU - G. Crescioli
AU - C. Ravaldi
AU - R. Bonaiuti
PY - 2026
DA - 2026/10/31
PB - medRxiv
DO - 10.64898/2026.10.31.26366978
UR - https://www.medrxiv.org/content/10.64898/2026.10.31.26366978v1
AB - Importance: One adult in 10 has consulted a chatbot for health information within 6 months, and nearly half of users do so to decide whether they need care. Referral regardless of need could add demand on stretched services. Objective: To compare referral to a professional by general-purpose language models in health and non-health situations against the level of contact that human raters judged necessary. Design, Setting, and Participants: Cross-sectional study of 4,346 responses generated in September 2026 by 53 general-purpose model configurations, released between March 2023 and September 2026 and accessed through APIs or run locally, to 82 investigator-written messages (41 health and 41 non-health). No patients were involved. Exposures: Health vs non-health scenario. Main Outcomes and Measures: Professional referral, unconditional or conditional on a future change or on doubt, coded by 3 language models and compared with the level of contact that physicians, pharmacists and 2 technically trained raters judged necessary. Results: Referral occurred in 74.7% of health and 46.3% of non-health responses, more often for health in every configuration (standardized difference, 28.3 percentage points; 95% CI, 18.3-37.0). Where raters judged contact unnecessary, 64.5% of health vs 27.6% of non-health responses recommended a professional (odds ratio, 17.08; 95% CI, 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%), and unconditional referral was 18.1% vs 6.6% (odds ratio, 4.66; 95% CI, 1.43-15.19). Where an in-person assessment was judged necessary, referral was similar (89.6% vs 84.5%). In medicines and physical health, referral was not associated with rated severity (odds ratio per point, 1.16; 95% CI, 0.53-2.55), unlike outside health (3.86; 95% CI, 1.79-8.33). The referral gap was 20.5 percentage points among configurations released in 2023 and 32.2 and 32.3 in 2025 and 2026 (4.8 percentage points per year; 95% CI, 2.3-7.2). Conclusions and Relevance: General-purpose language models referred people to a professional where a visit was needed, in health as outside it, but in health also in 2 in 3 situations that physicians and pharmacists judged manageable without contact, a pattern stronger in newer models. Evaluations of these systems should include whether their advice distinguishes situations that can be deferred or managed at a distance, alongside the referral of serious cases.
ER -