A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis.
6 affiliations
- 1Chair of Public Health and Health Services Research, IBE, LMU Medizin, LMU Munich, Munich, Germany. [email protected].
- 2Pettenkofer School of Public Health, Munich, Germany. [email protected].
- 3Chair of Public Health and Health Services Research, IBE, LMU Medizin, LMU Munich, Munich, Germany.
- 4Pettenkofer School of Public Health, Munich, Germany.
- 5Chair of Public Health Nutrition, University of Bayreuth, Bayreuth, Germany.
- 6Cognitive Interaction Technology Center (CITEC), Bielefeld University, Bielefeld, Germany.
Abstract
Background
For public health policy research, it is crucial to monitor policy debates to inform evidence-based policymaking. One popular method is to analyze the news for salient frames used to argue for or against a controversial topic. However, this method is time and resource-intensive since domain experts need to manually analyze relevant articles to identify frames. This study presents an approach for using large language models (LLMs) to extract frames and to systematically evaluate results.Methods
Building on the results of a framing analysis of the German media debate about a tax on animal-based foods, we developed a systematic workflow to utilize LLMs to extract all instances of the 10 most frequent manually coded negative frames from 46 news articles. With one prompt per article in a few-shot setting we generated a list of quote/frame pairs. We then evaluated results against human-coded ones and calculated precision, recall, and F1-score.Results
GPT-5 achieved 79.8% precision and 79.8% recall, identifying 75.2% of manually coded frames. Extracted quotes generally aligned with human-extracted text sections. Gemini 3.1 Pro Preview yielded higher precision but lower recall, resulting in a comparable F1-score (79.0% vs. 79.8% with GPT-5). These findings suggest the approach is generalizable across different frontier models.Conclusions
This study introduces a workflow for LLM-assisted frame extraction and transparent evaluation against an expert reference standard. Integrating this approach into public health policy coding can save resources and accelerate analysis. Ultimately, this could increase the representativeness and timeliness of data, enhancing our understanding of ongoing public health policy debates.
Related
- Framing anatomical variation in anatomy textbooks: Language use and its implications
- Utilizing the Contrastive Language-Image Pre-Training Model in Chest Radiography: Current Capabilities and Practical Guidance.
- Development of knowledge graphs for emergency events utilizing pre-trained language models
- Large Language Models in the Radiology Workflow: Current Applications, Challenges, and Future Perspectives.
- Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction
BibTeX
@article{wenderoth2026systematic,
title = {{A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis.}},
author = {Johanna Wenderoth and Peter von Philipsborn and Anna Leibinger and Philipp Cimiano},
journal = {Archives of public health = Archives belges de sante publique},
year = {2026},
month = oct,
doi = {10.1186/s13690-026-02089-6},
url = {https://europepmc.org/article/MED/42897673},
note = {CC BY},
}
RIS
TY - JOUR
TI - A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis.
AU - Johanna Wenderoth
AU - Peter von Philipsborn
AU - Anna Leibinger
AU - Philipp Cimiano
PY - 2026
DA - 2026/10/17
JO - Archives of public health = Archives belges de sante publique
DO - 10.1186/s13690-026-02089-6
UR - https://europepmc.org/article/MED/42897673
AB - Background For public health policy research, it is crucial to monitor policy debates to inform evidence-based policymaking. One popular method is to analyze the news for salient frames used to argue for or against a controversial topic. However, this method is time and resource-intensive since domain experts need to manually analyze relevant articles to identify frames. This study presents an approach for using large language models (LLMs) to extract frames and to systematically evaluate results.Methods Building on the results of a framing analysis of the German media debate about a tax on animal-based foods, we developed a systematic workflow to utilize LLMs to extract all instances of the 10 most frequent manually coded negative frames from 46 news articles. With one prompt per article in a few-shot setting we generated a list of quote/frame pairs. We then evaluated results against human-coded ones and calculated precision, recall, and F1-score.Results GPT-5 achieved 79.8% precision and 79.8% recall, identifying 75.2% of manually coded frames. Extracted quotes generally aligned with human-extracted text sections. Gemini 3.1 Pro Preview yielded higher precision but lower recall, resulting in a comparable F1-score (79.0% vs. 79.8% with GPT-5). These findings suggest the approach is generalizable across different frontier models.Conclusions This study introduces a workflow for LLM-assisted frame extraction and transparent evaluation against an expert reference standard. Integrating this approach into public health policy coding can save resources and accelerate analysis. Ultimately, this could increase the representativeness and timeliness of data, enhancing our understanding of ongoing public health policy debates.
SN - 0778-7367
SN - 2049-3258
ER -