Patient expectations for artificial intelligence transparency and agency: A qualitative study and proportionate framework for clinical artificial intelligence implementation
5 affiliations
- 1Hackensack Meridian School of Medicine, Nutley, New Jersey, United States of America
- 2Department of Medicine, Warren Alpert Medical School of Brown University, Providence, Rhode Island, United States of America
- 3Department of Urology, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States of America
- 4Hackensack Meridian Health, Edison, New Jersey, United States of America
- 5Bioethics Program, Columbia University, New York, New York, United States of America
Abstract
Artificial intelligence (AI) integration in healthcare is expanding rapidly, encompassing applications from ambient clinical scribes to diagnostic support tools. While AI adoption by physicians has recently doubled, research regarding patient expectations for transparency and agency remains limited. Current literature often fails to provide actionable guidance for translating patient perspectives into operational health system frameworks. This study utilizes feedback from Patient and Family Advisory Councils (PFACs) to explore community perspectives on healthcare AI and propose a transparency framework that engenders trust and prioritizes patient-centered care. In this qualitative study, 20 semi-structured interviews and intake surveys of PFAC members across nine hospital sites within a large healthcare network were conducted. Data analysis of interview transcripts employed a hybrid inductive and deductive thematic approach, utilizing Google’s Gemini 2.5 Pro for initial coding and theme generation, followed by rigorous human validation to reach interpretative consensus. Qualitative analysis identified four nested themes: 1) Ethical Prerequisites for AI Integration, 2) Systemic Prerequisites for Trust, 3) Preserving Clinician Oversight, and 4) Strategies for Meaningful Communication. Participants viewed transparency as an ethical “floor” necessary for autonomy and expressed that trust must be earned through evidence of systemic reliability and High Reliability Organization (HRO) principles. Critically, patients expect clinicians to serve as a “bridge” between technology and the patient, acting as the final arbiter of care decisions. These findings provide the foundation for a proportionate framework regarding communication that scales transparency based on clinical risk and the required level of patient agency. The framework categorizes AI engagement into three tiers: disclosure, assent, and consent. By nesting ethical and systemic foundations within actionable steps for clinician oversight and proactive communication, health systems can implement AI in a trust-engendering manner that preserves the sanctity of the patient-provider relationship. Author summary: Artificial intelligence is quickly becoming a part of everyday life, including medical care. However, while this new technology is being researched, explored, and implemented for its potential benefits, there has been little integration of patient opinion into current AI systems. In an effort to implement consultative AI, where all stakeholders’ perspectives are taken into account before model implementation, we wanted to understand what patients need to feel comfortable and respected when these technologies are used in their treatment. In order to do this, we spoke directly with patient advisory council members. Our findings show that patient trust is built on transparency and evidence of strong safety standards. The most common theme that was identified by participants was the need for transparency. Participants emphasized that being informed is the starting point that enables them to maintain control of their healthcare. Rooted in the findings from our data, we developed a three-level framework for communication. This system suggests that for low-risk tools, a simple notice is sufficient, while high-stakes decisions require a more detailed discussion and formal patient permission. We hope this work provides a roadmap for healthcare systems to embrace innovation while protecting the vital bond between patients and their doctors.
Related
- Family and informal caregivers’ acceptance, concerns and expectations regarding artificial intelligence-enabled dementia care: a protocol for qualitative evidence synthesis
- Transparency, accountability, and scholarly integrity: A systematic narrative review of artificial intelligence governance frameworks in healthcare publication
- Advancing antimicrobial stewardship in Africa through responsible artificial intelligence implementation
- As open as possible, as protected as necessary: Proportionate governance for environmental DNA under the BBNJ Agreement
- Toward Enactive Artificial Intelligence
BibTeX
@article{stein2026patient,
title = {{Patient expectations for artificial intelligence transparency and agency: A qualitative study and proportionate framework for clinical artificial intelligence implementation}},
author = {Guy Stein and Abinav R. Udaiyar and David Bouslov and Howard M. Ross and Charles E. Binkley},
journal = {PLOS Digital Health},
year = {2026},
month = nov,
doi = {10.1371/journal.pdig.0001816},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001816},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Patient expectations for artificial intelligence transparency and agency: A qualitative study and proportionate framework for clinical artificial intelligence implementation
AU - Guy Stein
AU - Abinav R. Udaiyar
AU - David Bouslov
AU - Howard M. Ross
AU - Charles E. Binkley
PY - 2026
DA - 2026/11/04
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001816
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001816
AB - Artificial intelligence (AI) integration in healthcare is expanding rapidly, encompassing applications from ambient clinical scribes to diagnostic support tools. While AI adoption by physicians has recently doubled, research regarding patient expectations for transparency and agency remains limited. Current literature often fails to provide actionable guidance for translating patient perspectives into operational health system frameworks. This study utilizes feedback from Patient and Family Advisory Councils (PFACs) to explore community perspectives on healthcare AI and propose a transparency framework that engenders trust and prioritizes patient-centered care. In this qualitative study, 20 semi-structured interviews and intake surveys of PFAC members across nine hospital sites within a large healthcare network were conducted. Data analysis of interview transcripts employed a hybrid inductive and deductive thematic approach, utilizing Google’s Gemini 2.5 Pro for initial coding and theme generation, followed by rigorous human validation to reach interpretative consensus. Qualitative analysis identified four nested themes: 1) Ethical Prerequisites for AI Integration, 2) Systemic Prerequisites for Trust, 3) Preserving Clinician Oversight, and 4) Strategies for Meaningful Communication. Participants viewed transparency as an ethical “floor” necessary for autonomy and expressed that trust must be earned through evidence of systemic reliability and High Reliability Organization (HRO) principles. Critically, patients expect clinicians to serve as a “bridge” between technology and the patient, acting as the final arbiter of care decisions. These findings provide the foundation for a proportionate framework regarding communication that scales transparency based on clinical risk and the required level of patient agency. The framework categorizes AI engagement into three tiers: disclosure, assent, and consent. By nesting ethical and systemic foundations within actionable steps for clinician oversight and proactive communication, health systems can implement AI in a trust-engendering manner that preserves the sanctity of the patient-provider relationship. Author summary: Artificial intelligence is quickly becoming a part of everyday life, including medical care. However, while this new technology is being researched, explored, and implemented for its potential benefits, there has been little integration of patient opinion into current AI systems. In an effort to implement consultative AI, where all stakeholders’ perspectives are taken into account before model implementation, we wanted to understand what patients need to feel comfortable and respected when these technologies are used in their treatment. In order to do this, we spoke directly with patient advisory council members. Our findings show that patient trust is built on transparency and evidence of strong safety standards. The most common theme that was identified by participants was the need for transparency. Participants emphasized that being informed is the starting point that enables them to maintain control of their healthcare. Rooted in the findings from our data, we developed a three-level framework for communication. This system suggests that for low-risk tools, a simple notice is sufficient, while high-stakes decisions require a more detailed discussion and formal patient permission. We hope this work provides a roadmap for healthcare systems to embrace innovation while protecting the vital bond between patients and their doctors.
SN - 2767-3170
ER -