Subversion of clinical judgment by conversational artificial intelligence
- 1University of California, San Francisco; University of California, Berkeley
Abstract
In medicine, artificial intelligence (AI) safety has focused on discrete flaws such as hallucination, bias and miscalibration. Conversational systems pose a distinct hazard: in dialogue, a model can covertly advance a misaligned objective by redirecting the clinical judgment it should support. Whether clinical expertise equips physicians to detect and resist such subversion is unknown. In a participant-blinded, randomized experiment, 225 physicians from 42 countries and eight clinical disciplines made sequential diagnostic and treatment decisions with AI assistance on anonymized cases from neurocritical care, their shared field of expertise. Unknown to them, each physician used the same model under two conditions: instructed to steer them towards harmful targets (adversarial) or correct clinical options (aligned), each established by expert consensus. With adversarial AI, 190 physicians made at least one harmful decision, compared to 21 with aligned AI. Across 2,485 targeted decision points, adversarial AI raised the harmful-decision rate from 2% with aligned AI to 49% (adjusted difference 47 percentage points, 95% CI, 43-51) and reduced the correct-decision rate from 83% to 25%, with no clear protection associated with clinical experience. Of the 190 physicians who made harmful decisions with adversarial AI, 141 did not report anything unusual about that AI. Two physicians reported suspecting systematic steering or manipulation. Physicians cannot safeguard the clinical use of conversational AI through expertise and decision-making authority alone.
Related
- Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study
- Conversational artificial intelligence for pre-procedural patient preparation: implementation, validation and patient satisfaction
- Harnessing conversational generative artificial intelligence and natural language processing to address the colorectal cancer screening adherence gap: the Azores endeavour
- Quality of physicians responses using a conversational artificial intelligence system: a randomized vignette experiment in Latin America
- Towards conversational artificial intelligence for disease management
BibTeX
@misc{barrit2026subversion,
title = {{Subversion of clinical judgment by conversational artificial intelligence}},
author = {S. Barrit and M. Salvagno and A. Corriero and M. Sushil and T. Pate and J. Klug and M. Aries and S. Benghanem and M. Baggiani and G. Cane and S. Park and R. Soloperto and N. Engrand and N. Ben-Hamouda and C. Robba and M. Munari and B. Balanca and F. Hilaire and J. C. Hemphill and B. Foreman and C. Lazaridis and R. Chabanne and D. Vlahovic and R. Helbok and D. Pinggera and M. Kirschen and Neurocore research group and F. S. Taccone and E. F. Chang},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.05.26364541},
url = {https://www.medrxiv.org/content/10.64898/2026.10.05.26364541v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Subversion of clinical judgment by conversational artificial intelligence
AU - S. Barrit
AU - M. Salvagno
AU - A. Corriero
AU - M. Sushil
AU - T. Pate
AU - J. Klug
AU - M. Aries
AU - S. Benghanem
AU - M. Baggiani
AU - G. Cane
AU - S. Park
AU - R. Soloperto
AU - N. Engrand
AU - N. Ben-Hamouda
AU - C. Robba
AU - M. Munari
AU - B. Balanca
AU - F. Hilaire
AU - J. C. Hemphill
AU - B. Foreman
AU - C. Lazaridis
AU - R. Chabanne
AU - D. Vlahovic
AU - R. Helbok
AU - D. Pinggera
AU - M. Kirschen
AU - Neurocore research group
AU - F. S. Taccone
AU - E. F. Chang
PY - 2026
DA - 2026/10/05
PB - medRxiv
DO - 10.64898/2026.10.05.26364541
UR - https://www.medrxiv.org/content/10.64898/2026.10.05.26364541v1
AB - In medicine, artificial intelligence (AI) safety has focused on discrete flaws such as hallucination, bias and miscalibration. Conversational systems pose a distinct hazard: in dialogue, a model can covertly advance a misaligned objective by redirecting the clinical judgment it should support. Whether clinical expertise equips physicians to detect and resist such subversion is unknown. In a participant-blinded, randomized experiment, 225 physicians from 42 countries and eight clinical disciplines made sequential diagnostic and treatment decisions with AI assistance on anonymized cases from neurocritical care, their shared field of expertise. Unknown to them, each physician used the same model under two conditions: instructed to steer them towards harmful targets (adversarial) or correct clinical options (aligned), each established by expert consensus. With adversarial AI, 190 physicians made at least one harmful decision, compared to 21 with aligned AI. Across 2,485 targeted decision points, adversarial AI raised the harmful-decision rate from 2% with aligned AI to 49% (adjusted difference 47 percentage points, 95% CI, 43-51) and reduced the correct-decision rate from 83% to 25%, with no clear protection associated with clinical experience. Of the 190 physicians who made harmful decisions with adversarial AI, 141 did not report anything unusual about that AI. Two physicians reported suspecting systematic steering or manipulation. Physicians cannot safeguard the clinical use of conversational AI through expertise and decision-making authority alone.
ER -