AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms
- 1Lane Fox Unit/Sleep Disorders Centre, Guy's and St Thomas' NHS Foundation Trust, London, UK
Abstract
BackgroundPatients increasingly consult free artificial intelligence (AI) chatbots about their symptoms, but these models may favour agreement with the patient over accuracy. In obstructive sleep apnoea (OSA), where most cases in the general population remain undiagnosed and diagnosis depends on referral, this could validate symptom denial. We tested whether consumer chatbots maintain guideline-concordant referral when a simulated patient minimises symptoms.
MethodsSeven guideline-concordant and standardised OSA vignettes (apnoea-hypopnoea-index, AHI 8-52 events/h; body mass index 28-42 kg/m2) were presented to five free chatbots (GPT-5, Gemini 2.5 Flash, Claude Sonnet 4.6, DeepSeek V4 Flash and Grok-4.3) under neutral and symptom-minimising conditions (10 simulated conversations per model- vignette-condition cell; N=700). Referral maintenance was scored by a structured rubric and every capitulation was human-confirmed.
ResultsAll chatbots maintained referral recommendations in neutral conversations (350/350; 100%). When simulated patients minimised symptoms and resisted referral, maintenance fell to 64.3% (225/350), a reduction of 35.7 percentage points (95% CI 5.5- 66.0; p=0.028). Rates ranged from 55.7% to 85.7% across chatbots, but evidence of between-chatbot differences was inconsistent across analyses. Rates differed across clinical vignettes (p=0.001). A variance analysis estimated greater variation between vignettes than between chatbots. In the three driving-risk vignettes, referral recommendations were abandoned in 46- 68% of symptom-minimising conversations, pooling all five chatbots within each vignette.
ConclusionsConsumer AI chatbots abandoned correct OSA referral recommendations in about one-third of symptom-minimising conversations, with higher observed abandonment in simulated cases involving driving risk. This may delay appropriate assessment and raises patient- and public-safety concerns.
Related
- Potential Use of AI Chatbots by Patients With Bowel Tumors: Opportunities and Implications for Low- and Middle-Income Countries.
- Clinical manifestations, genetic characteristics and laboratory markers of patients with epidermolysis bullosa: a single tertiary referral center experience
- Quality of life and associated factors among patients with cystic echinococcosis treated at referral hospitals in Northwestern Ethiopia
- Prevalence and associated factors of type 2 diabetes among HIV-infected patients at Arsi University Teaching and Referral Hospital, Southeast Ethiopia, 2023
- Performance of AI chatbots in responding to patients' frequently asked questions about laminate veneer restorations.
BibTeX
@misc{ratneswaran2026chatbots,
title = {{AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms}},
author = {D. Ratneswaran and S. Poirier and D. Hamman and M. Rykov and J. Steier},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.09.26365165},
url = {https://www.medrxiv.org/content/10.64898/2026.10.09.26365165v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - AI chatbots abandon sleep apnoea referral when simulated patients minimise symptoms
AU - D. Ratneswaran
AU - S. Poirier
AU - D. Hamman
AU - M. Rykov
AU - J. Steier
PY - 2026
DA - 2026/10/09
PB - medRxiv
DO - 10.64898/2026.10.09.26365165
UR - https://www.medrxiv.org/content/10.64898/2026.10.09.26365165v1
AB - BackgroundPatients increasingly consult free artificial intelligence (AI) chatbots about their symptoms, but these models may favour agreement with the patient over accuracy. In obstructive sleep apnoea (OSA), where most cases in the general population remain undiagnosed and diagnosis depends on referral, this could validate symptom denial. We tested whether consumer chatbots maintain guideline-concordant referral when a simulated patient minimises symptoms. MethodsSeven guideline-concordant and standardised OSA vignettes (apnoea-hypopnoea-index, AHI 8-52 events/h; body mass index 28-42 kg/m2) were presented to five free chatbots (GPT-5, Gemini 2.5 Flash, Claude Sonnet 4.6, DeepSeek V4 Flash and Grok-4.3) under neutral and symptom-minimising conditions (10 simulated conversations per model- vignette-condition cell; N=700). Referral maintenance was scored by a structured rubric and every capitulation was human-confirmed. ResultsAll chatbots maintained referral recommendations in neutral conversations (350/350; 100%). When simulated patients minimised symptoms and resisted referral, maintenance fell to 64.3% (225/350), a reduction of 35.7 percentage points (95% CI 5.5- 66.0; p=0.028). Rates ranged from 55.7% to 85.7% across chatbots, but evidence of between-chatbot differences was inconsistent across analyses. Rates differed across clinical vignettes (p=0.001). A variance analysis estimated greater variation between vignettes than between chatbots. In the three driving-risk vignettes, referral recommendations were abandoned in 46- 68% of symptom-minimising conversations, pooling all five chatbots within each vignette. ConclusionsConsumer AI chatbots abandoned correct OSA referral recommendations in about one-third of symptom-minimising conversations, with higher observed abandonment in simulated cases involving driving risk. This may delay appropriate assessment and raises patient- and public-safety concerns.
ER -