Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
- 1Department of Drug Design and Pharmacology, University of Copenhagen, Jagtvej 160, 2100 Copenhagen, Denmark
Abstract
Background: The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown.
Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018–2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies.
Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping.
Conclusion: Off-the-shelf general-purpose LLMs currently offered more reliable support for pharmacoepidemiologic study design than the smaller biomedical LLMs evaluated. In the model-controlled comparison (GPT-4o with Least-to-Most versus Active prompting), prompt strategy did not significantly affect performance (paired p = 0.93), and prompt comparisons are therefore reported descriptively. Because the two model groups also differed in scale and instruction-tuning, this contrast should be interpreted as a comparison of readily deployable options rather than of biomedical specialization alone.
Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1007/s11095-026-04190-5.
Related
- Towards Gender-Sensitive Health Data Infrastructures: Interoperability Challenges and Design Implications for Biomedical Engineering
- Advanced Nanoemulsion Architectures for Precision Theranostics and Next-Generation Biomedical Translation
- Advanced engineering strategies in human organoids for tissue repair and regeneration.
- Antiprotozoal drug exposure and reduced long-term mortality and morbidity: a multi-cohort pharmacoepidemiologic study
- Staging and Grading Advanced Periodontitis with Large Language Models from Standardized Case Summaries: A Multicenter Concordance and Reproducibility Study
BibTeX
@article{zhang2026employing,
title = {{Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design}},
author = {Xinyao Zhang and Nicole Sonne Heckmann and Manuela Del Castillo Suero and Francesco Paolo Speca and Maurizio Sessa},
journal = {Pharmaceutical Research},
year = {2026},
month = sep,
doi = {10.1007/s11095-026-04190-5},
url = {https://doi.org/10.1007/s11095-026-04190-5},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
AU - Xinyao Zhang
AU - Nicole Sonne Heckmann
AU - Manuela Del Castillo Suero
AU - Francesco Paolo Speca
AU - Maurizio Sessa
PY - 2026
DA - 2026/09/11
JO - Pharmaceutical Research
DO - 10.1007/s11095-026-04190-5
UR - https://doi.org/10.1007/s11095-026-04190-5
AB - Background: The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018–2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping. Conclusion: Off-the-shelf general-purpose LLMs currently offered more reliable support for pharmacoepidemiologic study design than the smaller biomedical LLMs evaluated. In the model-controlled comparison (GPT-4o with Least-to-Most versus Active prompting), prompt strategy did not significantly affect performance (paired p = 0.93), and prompt comparisons are therefore reported descriptively. Because the two model groups also differed in scale and instruction-tuning, this contrast should be interpreted as a comparison of readily deployable options rather than of biomedical specialization alone. Supplementary Information: The online version contains supplementary material available at https://doi.org/10.1007/s11095-026-04190-5.
SN - 0724-8741
SN - 1573-904X
ER -