When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications.
8 affiliations
- 1Department of Adult Intensive Care, Erasmus MC, Rotterdam, The Netherlands.
- 2Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands.
- 3Department of Clinical Physics and Informatics, Maasstad Hospital, Rotterdam, The Netherlands.
- 4Romion Health, Utrecht, The Netherlands.
- 5Dutch Patients' Federation, Utrecht, The Netherlands.
- 6Working Group on Digital Health, European Patients' Forum, Brussels, Belgium.
- 7Department of Risk Analysis for Products in Development, Netherlands Organization for Applied Scientific Research, The Hague, The Netherlands.
- 8Department of Medicine, University of Cambridge, Cambridge, UK.
Abstract
The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadvertently be used in ways that do not align with the intended purpose of the manufacturer, raising patient safety concerns and liability issues. We have identified a regulatory gap whereby LLM applications that could be repurposed, malfunction or be inadvertently misused for medical purposes fall outside the scope of the Medical Device Regulation (MDR) because they lack a medical intended purpose, and outside meaningful AI Act safety obligations because they do not qualify as high-risk systems, yet still present patient safety risks without adequate regulatory guidance. In this manuscript, we aim to address this regulatory gap for LLM applications in healthcare by identifying these ambiguous grey-area applications and providing four recommendations-validation, mitigation, compliance and alignment-focused on prevention and risk mitigation rather than reclassification as medical devices. These recommendations are intended to advance the responsible implementation of LLM applications in healthcare while ensuring patient safety.
Related
- Awareness and use of ecological and economic evaluation of medical devices in healthcare centers: a survey of French hospitals
- Artificial intelligence hallucinations in large language models for medical education and healthcare: a scoping review of detection, mitigation, and governance
- Navigating reconfiguration: case managers in public municipal healthcare - a scoping review protocol.
- Navigating result disclosure in the context of reproductive genetic carrier screening: an interview study of genetic healthcare professionals in Belgium
- Biofilm Formation on Invasive Medical Devices in Adult Intensive Care Units: A Scoping Review of Biomedical and Clinical Evidence
BibTeX
@article{workum2026when,
title = {{When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications.}},
author = {Jessica D Workum and Sade J Krijgsman and Ildiko Vajda and R M Van Stokkum and Jan van de Brand and Diederik Gommers and Ari Ercole and Michel van Genderen},
journal = {BMJ digital health \& AI},
year = {2026},
month = oct,
doi = {10.1136/bmjdh-2026-000003},
url = {https://europepmc.org/article/MED/42851737},
note = {CC BY-NC},
}
RIS
TY - JOUR
TI - When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications.
AU - Jessica D Workum
AU - Sade J Krijgsman
AU - Ildiko Vajda
AU - R M Van Stokkum
AU - Jan van de Brand
AU - Diederik Gommers
AU - Ari Ercole
AU - Michel van Genderen
PY - 2026
DA - 2026/10/07
JO - BMJ digital health & AI
DO - 10.1136/bmjdh-2026-000003
UR - https://europepmc.org/article/MED/42851737
AB - The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadvertently be used in ways that do not align with the intended purpose of the manufacturer, raising patient safety concerns and liability issues. We have identified a regulatory gap whereby LLM applications that could be repurposed, malfunction or be inadvertently misused for medical purposes fall outside the scope of the Medical Device Regulation (MDR) because they lack a medical intended purpose, and outside meaningful AI Act safety obligations because they do not qualify as high-risk systems, yet still present patient safety risks without adequate regulatory guidance. In this manuscript, we aim to address this regulatory gap for LLM applications in healthcare by identifying these ambiguous grey-area applications and providing four recommendations-validation, mitigation, compliance and alignment-focused on prevention and risk mitigation rather than reclassification as medical devices. These recommendations are intended to advance the responsible implementation of LLM applications in healthcare while ensuring patient safety.
SN - 3049-575X
ER -