Multimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa
- 1The Ohio State University
Abstract
BackgroundTimely cause-of-death (COD) information guides health policy, yet many deaths in low- and middle-income countries are not medically certified. In Africa, fewer than 10% of deaths are. Verbal autopsy (VA) yields a structured symptom questionnaire, and a free-text narrative. Mortality surveillance increasingly records the narrative alongside the questionnaire, but the algorithms used to produce VA-based statistics were designed for the structured responses only, so narratives inform physician COD assignment but do not contribute to algorithmic COD assignment. We evaluated whether supervised models could use both modalities to improve COD assignment.
MethodsWe analyzed 4,999 VA records for deaths at ages 12 years or older from a national sample of registered deaths in South Africa (RSA), using physician-assigned CODs as the reference standard. We trained unimodal and multimodal pipelines on the labelled records: pretrained language models (PLMs) fine-tuned on narratives alone, machine learning (ML) models on structured responses alone, and the two combined at the data, feature and decision levels. We also prompted a locally deployed generative large language model (LLM) on the same unimodal and multimodal inputs, without parameter updates. All were compared with InSilicoVA, the operational comparator representing current automated practice. We evaluated individual-level assignment, population-level agreement, calibration, and generalization across provinces in RSA, reporting these metrics rather than ranking pipelines on any one of them.
ResultsEvery supervised pipeline improved on InSilicoVA by 8.5-15.9 percentage points in top-1 accuracy and 3.5-7.1 in cause-specific mortality fraction (CSMF) accuracy. The narrative and structured pipelines performed similarly in aggregate but differed by cause: 63.9% of deaths received identical predictions, 11.6% were correct only from the narrative pipeline and 13.6% only from the questionnaire pipeline, with the narrative pipeline more accurate for noncommunicable causes and the questionnaire pipeline for communicable causes. Against the unimodal approaches, the decision-fusion pipelines added 4.2 to 5.4 percentage points in top-1 accuracy, concentrated in the most common causes, whereas feature fusion with cross-modal attention improved the less common causes instead, adding 6.6 (1.7-11.5) points in balanced accuracy and 5.5 (0.6-10.3) in macro F1. Feature fusion reached the highest CSMF accuracy (95.7%). Paired differences from the better unimodal representative ranged from -0.2 to +3.4 points, with all intervals including zero. The LLM exceeded InSilicoVA on individual-level metrics and reached the highest balanced accuracy of any pipeline (60.6%), while the supervised pipelines retained higher CSMF accuracy. Supervised pipelines had lower expected calibration error than InSilicoVA. Supervised gains held across all nine held-out provinces, including every top-1 comparison and nearly every macro F1 and CSMF comparison.
ConclusionsPLMs using narratives alone substantially improved on the current best practice and matched ML pipelines using structured responses, while multiple multimodal strategies produced further gains in individual COD assignment. The zero-shot LLM showed particular strength on rare causes, whereas supervised pipelines better recovered the population cause distribution. These findings support incorporating narratives as a core input to automated VA systems and evaluating representative fusion strategies for the intended output. External validation across settings and age groups will support broader implementation.
Related
- Toward integrated multimodal artificial intelligence in bone marrow examination: current state and open challenges
- Artificial Intelligence–Driven Multimodal Imaging for Cancer During Pregnancy: Advances in Maternal–Fetal Diagnostics and Precision Oncology
- Artificial Intelligence, Multi-Omics, and Multimodal Integration in Precision Radiotherapy for Head and Neck Cancer: From Biological Insights to Treatment Decisions
- Kaiser Permanente National Cross-Vendor Validation of Mammography Artificial Intelligence Computer-Aided Diagnosis Algorithms in a US-Representative Population
- Artificial Intelligence Use Among Health Care Professionals in the United States and United Kingdom: Cross-National Survey.
BibTeX
@misc{chu2026multimodal,
title = {{Multimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa}},
author = {Y. Chu and Z. Wu and T. H. McCormick and Z. R. Li and S. J. Clark},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.15.26364476},
url = {https://www.medrxiv.org/content/10.64898/2026.10.15.26364476v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Multimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa
AU - Y. Chu
AU - Z. Wu
AU - T. H. McCormick
AU - Z. R. Li
AU - S. J. Clark
PY - 2026
DA - 2026/10/15
PB - medRxiv
DO - 10.64898/2026.10.15.26364476
UR - https://www.medrxiv.org/content/10.64898/2026.10.15.26364476v1
AB - BackgroundTimely cause-of-death (COD) information guides health policy, yet many deaths in low- and middle-income countries are not medically certified. In Africa, fewer than 10% of deaths are. Verbal autopsy (VA) yields a structured symptom questionnaire, and a free-text narrative. Mortality surveillance increasingly records the narrative alongside the questionnaire, but the algorithms used to produce VA-based statistics were designed for the structured responses only, so narratives inform physician COD assignment but do not contribute to algorithmic COD assignment. We evaluated whether supervised models could use both modalities to improve COD assignment. MethodsWe analyzed 4,999 VA records for deaths at ages 12 years or older from a national sample of registered deaths in South Africa (RSA), using physician-assigned CODs as the reference standard. We trained unimodal and multimodal pipelines on the labelled records: pretrained language models (PLMs) fine-tuned on narratives alone, machine learning (ML) models on structured responses alone, and the two combined at the data, feature and decision levels. We also prompted a locally deployed generative large language model (LLM) on the same unimodal and multimodal inputs, without parameter updates. All were compared with InSilicoVA, the operational comparator representing current automated practice. We evaluated individual-level assignment, population-level agreement, calibration, and generalization across provinces in RSA, reporting these metrics rather than ranking pipelines on any one of them. ResultsEvery supervised pipeline improved on InSilicoVA by 8.5-15.9 percentage points in top-1 accuracy and 3.5-7.1 in cause-specific mortality fraction (CSMF) accuracy. The narrative and structured pipelines performed similarly in aggregate but differed by cause: 63.9% of deaths received identical predictions, 11.6% were correct only from the narrative pipeline and 13.6% only from the questionnaire pipeline, with the narrative pipeline more accurate for noncommunicable causes and the questionnaire pipeline for communicable causes. Against the unimodal approaches, the decision-fusion pipelines added 4.2 to 5.4 percentage points in top-1 accuracy, concentrated in the most common causes, whereas feature fusion with cross-modal attention improved the less common causes instead, adding 6.6 (1.7-11.5) points in balanced accuracy and 5.5 (0.6-10.3) in macro F1. Feature fusion reached the highest CSMF accuracy (95.7%). Paired differences from the better unimodal representative ranged from -0.2 to +3.4 points, with all intervals including zero. The LLM exceeded InSilicoVA on individual-level metrics and reached the highest balanced accuracy of any pipeline (60.6%), while the supervised pipelines retained higher CSMF accuracy. Supervised pipelines had lower expected calibration error than InSilicoVA. Supervised gains held across all nine held-out provinces, including every top-1 comparison and nearly every macro F1 and CSMF comparison. ConclusionsPLMs using narratives alone substantially improved on the current best practice and matched ML pipelines using structured responses, while multiple multimodal strategies produced further gains in individual COD assignment. The zero-shot LLM showed particular strength on rare causes, whereas supervised pipelines better recovered the population cause distribution. These findings support incorporating narratives as a core input to automated VA systems and evaluating representative fusion strategies for the intended output. External validation across settings and age groups will support broader implementation.
ER -