Evaluating large language models for lay summaries of radiology reports using tailored prompting strategies and mixed-method assessment
7 affiliations
- 1MSc eHealth Program, McMaster University, Hamilton, Ontario, Canada
- 2Department of Medical Imaging, McMaster University, Hamilton, Ontario, Canada
- 3Department of Diagnostic Imaging, Juravinski Hospital and Cancer Centre, Hamilton Health Sciences, Hamilton, Ontario, Canada
- 4Escarpment Cancer Research Institute, Hamilton Health Sciences and McMaster University, Hamilton, Ontario, Canada
- 5Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
- 6Department of Oncology, McMaster University, Juravinski Cancer Centre, Hamilton, Ontario, Canada
- 7CentRE for dAta science and digiTal hEalth (CREATE), Hamilton Health Sciences, Hamilton, Ontario, Canada
Abstract
Radiology reports are often filled with medical jargon that limits patient understanding. Lay summaries can improve understanding but are time-consuming for healthcare providers to create. The objective of this study is to explore the use of tailored prompts for five Large Language Models (LLMs) in generating lay summaries from radiology reports. Using 100 reports from the publicly available “BioNLP 2023 report summarization” dataset, lay summaries were generated by each LLM, under select prompting styles [Few-Shot (GPT-4), Generated Knowledge (GPT-4o mini, Gemini 1.5 – Pro, Gemini 1.5 – Flash), and Zero-Shot (Llama 3.1)] informed by a pilot work. The summaries were evaluated using a mixed-method framework: subjective assessment (Likert statements) by blinded experts (n = 2 radiology fellows) and Large Reasoning Models (LRMs) [(Gemini 2.5 – Pro (LRM 1); GPT-oss-120b (LRM 2)], and readability metrics (Flesch-Kincaid Grade Level and Flesch Reading Ease). Using percentage agreement of Likert statements, the LLM-prompt combinations’ performances were ranked, and Friedman and post-hoc Nemenyi tests were conducted. Gemini 1.5 - Flash and - Pro (generated knowledge) were rated highest by human experts and LRMs for generating actionable lay summaries that require minimal supervision [P < 4.97 × 10-2 (Rater 1); P < 9.03 × 10-21 (Rater 2), P < 6.90 × 10-15 (LRM 1), P < 2.760 × 10-5 (LRM 2). GPT-4 (few-shot) achieved the highest human-rated accuracy (98%), while Gemini 1.5 – Flash (LRM 1-rated: 95%) and Gemini 1.5 – Pro (LRM 2-rated: 91%) ranked first in LRM-rated accuracy. Gemini 1.5 - Pro produced the most accessible summaries (Flesch-Kincaid Grade Level: 7.55 ± 1.38, Flesch Reading Ease: 67.84 ± 7.78). Strong agreement was observed between experts and LRMs [0.96% (LRM 1) and 3.4% (LRM 2) complete disagreement]. Overall, this study highlights Gemini-models with generated knowledge prompts and the potential of LRM evaluators in assessing LLM-generated lay summaries. Author summary: Creating plain-language medical summaries for patients is important, but it adds extra work for healthcare providers. While previously Artificial Intelligence systems, Large Language Models (LLM), were used to create patient-friendly summaries, this study is the first to explore how tailored prompt styles affect the model performance in generating summaries from radiology reports. Five models were prompted using pilot study-informed tailored styles: 1. What the model “knows” (Generated knowledge prompt for Gemini 1.5 - Flash, Gemini 1.5 – Pro and GPT-4o mini), 2. Guided examples (Few-shot for GPT-4) and 3. Only instructions (Zero-shot for Llama 3.1), to generate summaries for patients. These summaries were evaluated by human experts (radiology fellows), Gemini 2.5 – Pro and GPT-oss-120b [Large Reasoning Models (LRM)] and reading metrics. The results show that Gemini 1.5 – Flash and Gemini 1.5 – Pro with Generated knowledge prompting produced 7th-9th grade level summaries that had actionable and beneficial information and were more trusted by experts. We found that GPT-4 with Few-shot prompting produced the most accurate lay summaries according to the human raters. There was a strong similarity in ratings between the human experts and LRM evaluators. These findings show the potential of using AI to generate and evaluate clinical text summarization.
Related
- No significant differences between large language model prompting strategies and human experts in summarisation of 'the formal EU-US meniscus rehabilitation consensus: An ESSKA-AOSSM-AASPT Initiative'.
- SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies
- OVER-TIME II trial: design and rationale of a multicentre randomized study evaluating antithrombotic strategies in patients with acute coronary syndrome and coronary artery ectasia
- Evaluating the effect of class closure strategies on norovirus outbreaks in educational settings
- Evaluating Multiple Treatments and Strategies for Sepsis in Adaptive Platform (EURASSIA) trial in Japan: a core protocol.
BibTeX
@article{tasneem2026evaluating,
title = {{Evaluating large language models for lay summaries of radiology reports using tailored prompting strategies and mixed-method assessment}},
author = {Nanziba Tasneem and Christian B. van der Pol and Ambreen Zahoor and Nitin Juggath and Kyle McGowan and Cynthia Lokker and Ashirbani Saha},
journal = {PLOS Digital Health},
year = {2026},
month = sep,
doi = {10.1371/journal.pdig.0001672},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001672},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Evaluating large language models for lay summaries of radiology reports using tailored prompting strategies and mixed-method assessment
AU - Nanziba Tasneem
AU - Christian B. van der Pol
AU - Ambreen Zahoor
AU - Nitin Juggath
AU - Kyle McGowan
AU - Cynthia Lokker
AU - Ashirbani Saha
PY - 2026
DA - 2026/09/17
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001672
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001672
AB - Radiology reports are often filled with medical jargon that limits patient understanding. Lay summaries can improve understanding but are time-consuming for healthcare providers to create. The objective of this study is to explore the use of tailored prompts for five Large Language Models (LLMs) in generating lay summaries from radiology reports. Using 100 reports from the publicly available “BioNLP 2023 report summarization” dataset, lay summaries were generated by each LLM, under select prompting styles [Few-Shot (GPT-4), Generated Knowledge (GPT-4o mini, Gemini 1.5 – Pro, Gemini 1.5 – Flash), and Zero-Shot (Llama 3.1)] informed by a pilot work. The summaries were evaluated using a mixed-method framework: subjective assessment (Likert statements) by blinded experts (n = 2 radiology fellows) and Large Reasoning Models (LRMs) [(Gemini 2.5 – Pro (LRM 1); GPT-oss-120b (LRM 2)], and readability metrics (Flesch-Kincaid Grade Level and Flesch Reading Ease). Using percentage agreement of Likert statements, the LLM-prompt combinations’ performances were ranked, and Friedman and post-hoc Nemenyi tests were conducted. Gemini 1.5 - Flash and - Pro (generated knowledge) were rated highest by human experts and LRMs for generating actionable lay summaries that require minimal supervision [P < 4.97 × 10-2 (Rater 1); P < 9.03 × 10-21 (Rater 2), P < 6.90 × 10-15 (LRM 1), P < 2.760 × 10-5 (LRM 2). GPT-4 (few-shot) achieved the highest human-rated accuracy (98%), while Gemini 1.5 – Flash (LRM 1-rated: 95%) and Gemini 1.5 – Pro (LRM 2-rated: 91%) ranked first in LRM-rated accuracy. Gemini 1.5 - Pro produced the most accessible summaries (Flesch-Kincaid Grade Level: 7.55 ± 1.38, Flesch Reading Ease: 67.84 ± 7.78). Strong agreement was observed between experts and LRMs [0.96% (LRM 1) and 3.4% (LRM 2) complete disagreement]. Overall, this study highlights Gemini-models with generated knowledge prompts and the potential of LRM evaluators in assessing LLM-generated lay summaries. Author summary: Creating plain-language medical summaries for patients is important, but it adds extra work for healthcare providers. While previously Artificial Intelligence systems, Large Language Models (LLM), were used to create patient-friendly summaries, this study is the first to explore how tailored prompt styles affect the model performance in generating summaries from radiology reports. Five models were prompted using pilot study-informed tailored styles: 1. What the model “knows” (Generated knowledge prompt for Gemini 1.5 - Flash, Gemini 1.5 – Pro and GPT-4o mini), 2. Guided examples (Few-shot for GPT-4) and 3. Only instructions (Zero-shot for Llama 3.1), to generate summaries for patients. These summaries were evaluated by human experts (radiology fellows), Gemini 2.5 – Pro and GPT-oss-120b [Large Reasoning Models (LRM)] and reading metrics. The results show that Gemini 1.5 – Flash and Gemini 1.5 – Pro with Generated knowledge prompting produced 7th-9th grade level summaries that had actionable and beneficial information and were more trusted by experts. We found that GPT-4 with Few-shot prompting produced the most accurate lay summaries according to the human raters. There was a strong similarity in ratings between the human experts and LRM evaluators. These findings show the potential of using AI to generate and evaluate clinical text summarization.
SN - 2767-3170
ER -