Artificial Intelligence for vaccine communication: comparing main available Large Language Models
Abstract
Background: Vaccination is one of the most powerful public health interventions. Large language models (LLMs) have demonstrated strong potential in delivering health-related information. This study compared the performance of ChatGPT 3.5 (G3E), ChatGPT 4o in English (G4E), Claude 3.0 (CDE), Gemini 1.5 (GME), and ChatGPT 4o in Italian (G4I) regarding vaccination communication.
Methods: A mixed-method study was performed. 26 realistic healthcare scenarios were created, then rated via a modified DISCERN questionnaire by 4 independent experts as 1-5 Likert scores. Information Reliability, Information Quality, Medical Appropriateness, Impact on Vaccine Hesitancy, Potential for Behavioral Influence, and Overall Rating were evaluated.
Results: Inter-rater agreement was substantial (Gwet’s AC2 = 0.061). Effects were reported as log(OR), with value 0 indicating no difference. G4E demonstrated the highest performance, with log(OR) = 0.77 (95% CI: 0.18–1.37, p = 0.010) for Overall Rating and log(OR) = 0.62 (95% CI: 0.04–1.20, p = 0.036) for Medical Appropriateness, indicating higher odds of better ratings (log(OR) = 0 means no difference). G4I showed comparatively improved performance in Information Quality (log(OR) = 0.57, 95% CI: 0.06–1.08, p = 0.030), although it performed less favorably in domains related to vaccine hesitancy and behavioral influence. GME exhibited weaker performance across qualitative dimensions, whereas CDE and G3E produced intermediate and relatively consistent results.
Conclusions: Differences in LLM performance likely depend on model architecture, training data, and language-specific optimization. Overall, LLMs display heterogeneous performance in delivering accurate and behaviorally impactful vaccination information, highlighting the need for careful evaluation prior to health communication strategies implementation. These findings highlight the importance of domain-specific adaptations, culturally-sensitive data curation and human expert supervision.
Key messages: • LLMs differ markedly in vaccine communication, depending on the evaluation domain, highlighting the need for careful model selection in public health messaging.
• AI-enabled vaccine communication requires language adaptation, cultural sensitivity, and expert oversight to ensure accuracy and positive behavioral impact.
Related
- Theory of Artificial Intelligence Adaptation for Linguistic Concordance: Bridging Gaps in Patient-Centered Communication
- The Influence of Large Language Model-Generated Artificial Intelligence Risk Estimates on Clinical Decision-Making in Breast Imaging: A Simulation-Based Study
- Artificial Intelligence Scribes and Same-Day Note Closure by Patient Language Preference and Interpreter Modality
- Exploring nursing students' communication experiences with artificial intelligence-supported virtual patients: a qualitative study.
- Calibrating trust in generative artificial intelligence for clinical communication through Chinese medical humanism
BibTeX
@article{pumpo2026artificial,
title = {{Artificial Intelligence for vaccine communication: comparing main available Large Language Models}},
author = {M Di Pumpo and L Villani and V Maio and MR Gualano and E Marziali and L De Maio and R Mancini and S Boccia and P Laurenti and W Ricciardi},
journal = {The European Journal of Public Health},
year = {2026},
month = oct,
doi = {10.1093/eurpub/ckag169.134},
url = {https://doi.org/10.1093/eurpub/ckag169.134},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Artificial Intelligence for vaccine communication: comparing main available Large Language Models
AU - M Di Pumpo
AU - L Villani
AU - V Maio
AU - MR Gualano
AU - E Marziali
AU - L De Maio
AU - R Mancini
AU - S Boccia
AU - P Laurenti
AU - W Ricciardi
PY - 2026
DA - 2026/10/01
JO - The European Journal of Public Health
DO - 10.1093/eurpub/ckag169.134
UR - https://doi.org/10.1093/eurpub/ckag169.134
AB - Background: Vaccination is one of the most powerful public health interventions. Large language models (LLMs) have demonstrated strong potential in delivering health-related information. This study compared the performance of ChatGPT 3.5 (G3E), ChatGPT 4o in English (G4E), Claude 3.0 (CDE), Gemini 1.5 (GME), and ChatGPT 4o in Italian (G4I) regarding vaccination communication. Methods: A mixed-method study was performed. 26 realistic healthcare scenarios were created, then rated via a modified DISCERN questionnaire by 4 independent experts as 1-5 Likert scores. Information Reliability, Information Quality, Medical Appropriateness, Impact on Vaccine Hesitancy, Potential for Behavioral Influence, and Overall Rating were evaluated. Results: Inter-rater agreement was substantial (Gwet’s AC2 = 0.061). Effects were reported as log(OR), with value 0 indicating no difference. G4E demonstrated the highest performance, with log(OR) = 0.77 (95% CI: 0.18–1.37, p = 0.010) for Overall Rating and log(OR) = 0.62 (95% CI: 0.04–1.20, p = 0.036) for Medical Appropriateness, indicating higher odds of better ratings (log(OR) = 0 means no difference). G4I showed comparatively improved performance in Information Quality (log(OR) = 0.57, 95% CI: 0.06–1.08, p = 0.030), although it performed less favorably in domains related to vaccine hesitancy and behavioral influence. GME exhibited weaker performance across qualitative dimensions, whereas CDE and G3E produced intermediate and relatively consistent results. Conclusions: Differences in LLM performance likely depend on model architecture, training data, and language-specific optimization. Overall, LLMs display heterogeneous performance in delivering accurate and behaviorally impactful vaccination information, highlighting the need for careful evaluation prior to health communication strategies implementation. These findings highlight the importance of domain-specific adaptations, culturally-sensitive data curation and human expert supervision. Key messages: • LLMs differ markedly in vaccine communication, depending on the evaluation domain, highlighting the need for careful model selection in public health messaging. • AI-enabled vaccine communication requires language adaptation, cultural sensitivity, and expert oversight to ensure accuracy and positive behavioral impact.
SN - 1101-1262
SN - 1464-360X
ER -