Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)
- 1LiGHT Lab, Swiss Federal Institute of Technology Lausanne (EPFL), Lausanne, Switzerland
Abstract
IntroductionLarge language models (LLMs) have the potential to strengthen clinical decision-making in low-resource primary healthcare (PHC) settings. However, most LLMs are developed and benchmarked in high-resource settings and evidence on their safety and contextual appropriateness in Sub-Saharan Africa remains limited. The hypMOOVE-PHC study is the hypothetical vignette phase of the Massive Open Online Validation and Evaluation (MOOVE) initiative, implemented in Kenya, Malawi, and Tanzania. It aims to validate a pool of LLMs through clinical review of expert-generated vignettes.
Methods and analysisThis is a fully crossed repeated-measures comparative evaluation study. In each country, experienced clinicians develop 200-250 hypothetical clinical vignettes reflecting realistic patient presentations and independently produce a human benchmark care plan for each. Vignettes are used to prompt a selection of six open-source and proprietary LLMs selected based on code availability, local hostability, and model size. During in-person workshops (valiDATAthons), independent clinical experts rate LLM- and human-generated responses in source-attribution masked side-by-side comparisons across five dimensions (clinical soundness, safety, contextual fit, clarity & completeness, and appropriate confidence). The primary endpoints are each LLMs overall performance profile and non-inferior safety profile, as compared to the human benchmark. At minimum, 358 evaluations per LLM (or 1,253 paired evaluations in total) are required per country.
Ethics and disseminationThe study is approved by the EPFL Human Ethics Research Committee in Switzerland, Harvard T.H. Chan School of Public Health in the USA, KNH-UoN Ethics and Research Committee in Kenya, MUBAS Research Ethics Committee in Malawi, and MUHAS Research and Ethics Committee and National Institute for Medical Research in Tanzania. Findings will be reported according to the TRIPOD-LLM framework and shared with national ministries of health, disseminated at conferences and in peer-reviewed journals, and de-identified benchmark data will be released under FAIR principles.
Strengths and limitations of this studyO_LIThe hypMOOVE evaluation uses guideline-based expert-validated vignettes, and responses are scored by practicing clinicians with contextual knowledge. C_LIO_LIIt evaluates a prospectively qualified pool of both open-source and proprietary LLMs that are judged against common criteria. C_LIO_LIInvolvement of three countries allows for cross-country comparison. C_LIO_LIThe evaluation is source-attribution masked: evaluators are not told which human or model produced each response, but distinctively human linguistic cues could reveal human authorship, creating a risk of source-type leakage and functional unblinding of the human label. C_LIO_LIAs a hypothetical-vignette phase, hypMOOVE findings describe LLM behavior on constructed scenarios and cannot establish real-world safety or effectiveness; this is addressed by the subsequent prospective phases outside the scope of this protocol. C_LI
Related
- Reclassifying Ablation Energy Risk Profiles in Atrial Fibrillation: A Narrative-Adjudicated Comparative Analysis of the MAUDE Database
- Developing and evaluating a customized evidence-based safety training program for healthcare staff: a quality improvement approach
- Development and content validation of a risk-stratified nursing intervention programme for the prevention of healthcare-associated infections
- Patient expectations and healthcare professional education in obesity care: A comparative qualitative study
- Why do patients bypass primary care? Trust, resource concentration, and risk in China's tiered healthcare system: A comparative policy perspective
BibTeX
@misc{macharia2026evaluating,
title = {{Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)}},
author = {P. Macharia and C. Kachimanga and M. K. Mahende and E. Mapunda and Y. Sonnay and A. Ntracha and S. Woodbury and I. Inwani and F. Otieno and H. R. Sawe and S. Kumwenda and G. Mhalu and W. Essuman and F. Elhassan and B. Duron and M. Henriet and O. Z. Azgaoui and V. Rousset and P. Ahumuza and K. S. Foer and S. Lipsitz and L. Klein and A. Kulinkina and K. Keitel and M.-A. Hartley},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.14.26362984},
url = {https://www.medrxiv.org/content/10.64898/2026.09.14.26362984v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)
AU - P. Macharia
AU - C. Kachimanga
AU - M. K. Mahende
AU - E. Mapunda
AU - Y. Sonnay
AU - A. Ntracha
AU - S. Woodbury
AU - I. Inwani
AU - F. Otieno
AU - H. R. Sawe
AU - S. Kumwenda
AU - G. Mhalu
AU - W. Essuman
AU - F. Elhassan
AU - B. Duron
AU - M. Henriet
AU - O. Z. Azgaoui
AU - V. Rousset
AU - P. Ahumuza
AU - K. S. Foer
AU - S. Lipsitz
AU - L. Klein
AU - A. Kulinkina
AU - K. Keitel
AU - M.-A. Hartley
PY - 2026
DA - 2026/09/14
PB - medRxiv
DO - 10.64898/2026.09.14.26362984
UR - https://www.medrxiv.org/content/10.64898/2026.09.14.26362984v1
AB - IntroductionLarge language models (LLMs) have the potential to strengthen clinical decision-making in low-resource primary healthcare (PHC) settings. However, most LLMs are developed and benchmarked in high-resource settings and evidence on their safety and contextual appropriateness in Sub-Saharan Africa remains limited. The hypMOOVE-PHC study is the hypothetical vignette phase of the Massive Open Online Validation and Evaluation (MOOVE) initiative, implemented in Kenya, Malawi, and Tanzania. It aims to validate a pool of LLMs through clinical review of expert-generated vignettes. Methods and analysisThis is a fully crossed repeated-measures comparative evaluation study. In each country, experienced clinicians develop 200-250 hypothetical clinical vignettes reflecting realistic patient presentations and independently produce a human benchmark care plan for each. Vignettes are used to prompt a selection of six open-source and proprietary LLMs selected based on code availability, local hostability, and model size. During in-person workshops (valiDATAthons), independent clinical experts rate LLM- and human-generated responses in source-attribution masked side-by-side comparisons across five dimensions (clinical soundness, safety, contextual fit, clarity & completeness, and appropriate confidence). The primary endpoints are each LLMs overall performance profile and non-inferior safety profile, as compared to the human benchmark. At minimum, 358 evaluations per LLM (or 1,253 paired evaluations in total) are required per country. Ethics and disseminationThe study is approved by the EPFL Human Ethics Research Committee in Switzerland, Harvard T.H. Chan School of Public Health in the USA, KNH-UoN Ethics and Research Committee in Kenya, MUBAS Research Ethics Committee in Malawi, and MUHAS Research and Ethics Committee and National Institute for Medical Research in Tanzania. Findings will be reported according to the TRIPOD-LLM framework and shared with national ministries of health, disseminated at conferences and in peer-reviewed journals, and de-identified benchmark data will be released under FAIR principles. Strengths and limitations of this studyO_LIThe hypMOOVE evaluation uses guideline-based expert-validated vignettes, and responses are scored by practicing clinicians with contextual knowledge. C_LIO_LIIt evaluates a prospectively qualified pool of both open-source and proprietary LLMs that are judged against common criteria. C_LIO_LIInvolvement of three countries allows for cross-country comparison. C_LIO_LIThe evaluation is source-attribution masked: evaluators are not told which human or model produced each response, but distinctively human linguistic cues could reveal human authorship, creating a risk of source-type leakage and functional unblinding of the human label. C_LIO_LIAs a hypothetical-vignette phase, hypMOOVE findings describe LLM behavior on constructed scenarios and cannot establish real-world safety or effectiveness; this is addressed by the subsequent prospective phases outside the scope of this protocol. C_LI
ER -