A mixed-methods evaluation of a language model-powered breast cancer note augmentation tool for patients
8 affiliations
- 1Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, 02139, MA, USA
- 2University of Michigan, 2260 Hayward St, Ann Arbor, 48109, MI, USA
- 3Division of Hematology and Oncology, Department of Medicine, University of Washington, 825 Eastlake Avenue East, Seattle, 98109, WA, USA
- 4Microsoft Research, 14820 NE 36th St, Redmond, 98052, WA, USA
- 5Massachusetts General Hospital, 55 Fruit St, Boston, 02114, MA, USA
- 6School of Medicine, Stanford University, 291 Campus Drive, Stanford, 94305, CA, USA
- 7Department of Oncology, Mayo Clinic, 5777 E. Mayo Blvd., Phoenix, 85054, AZ, USA
- 8Duke University, 2424 Erwin Rd, Durham, 27708, NC, USA
Abstract
Large language models (LLMs) have the potential to make clinical notes more understandable to patients, but tools must be tested with patients pre-deployment. We developed a patient-facing tool that uses an LLM (GPT-4-031) to generate “augmentations” to notes: simplified note text, defined medical terminology, frequently asked questions, extracted key information, and to-do items. We evaluated the tool with 200 healthy female participants and 8 notes. Compared to control, participants with access to augmentations had greater self-reported confidence (2.83 ± 0.05 augmentations, 2.18 ± 0.07 control, p<0.001) and scored higher on comprehension questions (0.63 ± 0.02 augmentations, 0.54 ± 0.04 control, p<0.001). We conducted interviews with 7 participants with a history of breast cancer and their own notes who described ease of use and perceived improvements in note understanding, but also reported concerns around LLMs. Three clinicians conducted an error analysis and found most errors to be minor and related to assumed context.
Related
- Evaluation of an AI-Powered Patient Education Application in Bariatric Surgery: A Prospective Mixed-Methods Feasibility Study
- Development and expert evaluation of a culturally adapted large language model for Malay clinical history-taking simulation: mixed methods study
- Cross-Cultural Adaptation and Initial Psychometric Evaluation of the Kazakh-Language Sino-Nasal Outcome Test-22 (SNOT-22) in Patients with Chronic Rhinosinusitis with Nasal Polyps
- Guideline adherence and expert-rated potential safety concerns in AI-generated dietary advice for patients with cirrhosis: a comparative evaluation of large language models.
- Development and evaluation effectiveness of interprofessional service package for promoting social participation of patients with Severe Mental Disorders (BARAMA): A protocol for mixed methods study.
BibTeX
@article{mannhardt2026mixedmethods,
title = {{A mixed-methods evaluation of a language model-powered breast cancer note augmentation tool for patients}},
author = {Niklas Mannhardt and Elizabeth Bondi and Barbara D. Lam and Hussein Mozannar and Chloe P. O’Connell and Mercy Asiedu and Alejandro Buendia and Tatiana Urman and Irbaz Bin Riaz and Catherine Ricciardi and Monica Agrawal and Marzyeh Ghassemi and David Alexander Sontag},
journal = {Scientific Reports},
year = {2026},
month = nov,
doi = {10.1038/s41598-026-59948-z},
url = {https://www.nature.com/articles/s41598-026-59948-z},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - A mixed-methods evaluation of a language model-powered breast cancer note augmentation tool for patients
AU - Niklas Mannhardt
AU - Elizabeth Bondi
AU - Barbara D. Lam
AU - Hussein Mozannar
AU - Chloe P. O’Connell
AU - Mercy Asiedu
AU - Alejandro Buendia
AU - Tatiana Urman
AU - Irbaz Bin Riaz
AU - Catherine Ricciardi
AU - Monica Agrawal
AU - Marzyeh Ghassemi
AU - David Alexander Sontag
PY - 2026
DA - 2026/11/05
JO - Scientific Reports
DO - 10.1038/s41598-026-59948-z
UR - https://www.nature.com/articles/s41598-026-59948-z
AB - Large language models (LLMs) have the potential to make clinical notes more understandable to patients, but tools must be tested with patients pre-deployment. We developed a patient-facing tool that uses an LLM (GPT-4-031) to generate “augmentations” to notes: simplified note text, defined medical terminology, frequently asked questions, extracted key information, and to-do items. We evaluated the tool with 200 healthy female participants and 8 notes. Compared to control, participants with access to augmentations had greater self-reported confidence (2.83 ± 0.05 augmentations, 2.18 ± 0.07 control, p<0.001) and scored higher on comprehension questions (0.63 ± 0.02 augmentations, 0.54 ± 0.04 control, p<0.001). We conducted interviews with 7 participants with a history of breast cancer and their own notes who described ease of use and perceived improvements in note understanding, but also reported concerns around LLMs. Three clinicians conducted an error analysis and found most errors to be minor and related to assumed context.
SN - 2045-2322
ER -