Human evaluators vs. LLM-as-a-Judge: toward scalable evaluation of GenAI in global health
4 affiliations
- 1PATH, London, UK
- 2Digital Umuganda, Kigali, Rwanda
- 3PATH, London, UK. [email protected]
- 4University of Birmingham, Birmingham, UK. [email protected]
Abstract
Evaluating generative AI output remains a critical bottleneck for safe and scalable deployment of AI in healthcare. Expert clinical judgement is often presented as the gold standard, but human assessment is costly and inconsistent. LLM-as-judge systems, i.e., leveraging AI to evaluate other AI outputs, have been proposed, yet their reliability in global health remains untested. We compared five LLM judges and six human clinicians in evaluating responses to questions posed by Rwandan health workers. The highest-performing LLM-judge (Claude-4.1-Opus) matched human evaluators on only four of eleven evaluation criteria, with other models scoring too leniently (Gemini-2.5-Pro) or too harshly (GPT-5). Constructing LLM-juries to balance model-specific biases improved agreement on only one additional criterion. Notably, performance and cost-effectiveness fell when moving from English to Kinyarwanda. Overall, while LLM-judges show promise, their inability to handle linguistic and cultural context is a critical limitation, underscoring the need for further investment in scalable evaluation solutions.
Related
- AI Video Analysis of Psychomotor Performance in EMS Education: Agreement With Human Evaluators Across Three Skills
- Impacts of long-term climate change on human health: a global scoping review of quantitative research
- Scalable Causal-Interpretable Machine Learning for Cancer Prescreening Using Electronic Health Records
- Global registry of alopecia areata disease severity and treatment safety: findings from an international pilot and optimization of a scalable real-world registry
- Low-intensity mind–body exercise for mental health in ageing societies: Tai Chi as a scalable intervention across depression, anxiety, insomnia, loneliness, and cognitive decline
BibTeX
@article{williams2026human,
title = {{Human evaluators vs. LLM-as-a-Judge: toward scalable evaluation of GenAI in global health}},
author = {Gwydion Williams and Samuel Rutunda and Floris Nzabakira and Bilal Akhter Mateen},
journal = {npj Digital Medicine},
year = {2026},
month = jul,
doi = {10.1038/s41746-026-02992-w},
url = {https://doi.org/10.1038/s41746-026-02992-w},
note = {CC BY},
}
RIS
TY - JOUR
TI - Human evaluators vs. LLM-as-a-Judge: toward scalable evaluation of GenAI in global health
AU - Gwydion Williams
AU - Samuel Rutunda
AU - Floris Nzabakira
AU - Bilal Akhter Mateen
PY - 2026
DA - 2026/07/20
JO - npj Digital Medicine
DO - 10.1038/s41746-026-02992-w
UR - https://doi.org/10.1038/s41746-026-02992-w
AB - Evaluating generative AI output remains a critical bottleneck for safe and scalable deployment of AI in healthcare. Expert clinical judgement is often presented as the gold standard, but human assessment is costly and inconsistent. LLM-as-judge systems, i.e., leveraging AI to evaluate other AI outputs, have been proposed, yet their reliability in global health remains untested. We compared five LLM judges and six human clinicians in evaluating responses to questions posed by Rwandan health workers. The highest-performing LLM-judge (Claude-4.1-Opus) matched human evaluators on only four of eleven evaluation criteria, with other models scoring too leniently (Gemini-2.5-Pro) or too harshly (GPT-5). Constructing LLM-juries to balance model-specific biases improved agreement on only one additional criterion. Notably, performance and cost-effectiveness fell when moving from English to Kinyarwanda. Overall, while LLM-judges show promise, their inability to handle linguistic and cultural context is a critical limitation, underscoring the need for further investment in scalable evaluation solutions.
SN - 2398-6352
ER -