Large Language Model-Assisted Semiautomated Data Harmonization Pipeline for the Multiple Chronic Disease Disparities Research Consortium: Framework Development and Comparative Validation Study.
11 affiliations
- 1Department of Epidemiology and Biostatistics, University of California, 550 16th St., Floor 2, San Francisco, CA, 94143, United States.
- 2Division of Preventive Science, Department of Medicine, University of California, San Francisco, CA, United States.
- 3Research Coordinating Center for Reducing Disparities in Multiple Chronic Diseases, University of California, San Francisco, CA, United States.
- 4Computational Precision Health, University of California San Francisco and University of California Berkeley, Berkeley, CA, United States.
- 5Social and Behavioral Sciences Branch, Division of Intramural Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, BG 6710 Rockledge Dr Wing B, Bethesda, MD, 20817, United States, 1 301-555-6677.
- 6Division of Oral Epidemiology & Dental Public Health, University of California, San Francisco, CA, United States.
- 7Pharmaceutical Science and Pharmacogenomics program, University of California, San Francisco, CA, United States.
- 8Bakar Computational Health Sciences Institute, University of California, San Francisco, CA, United States.
- 9College of Computer Mathematical and Natural Science, Department of Epidemiology and Biostatistics, School of Public Health, The Artificial Intelligence Interdisciplinary Institute at Maryland (AIM), University System of Maryland, College Park, MD, United States.
- 10Francis I. Proctor Foundation, University of California, San Francisco, CA, United States.
- 11Massey Comprehensive Cancer Center, Virginia Commonwealth University, Richmond, VA, United States.
Abstract
Background
The increasing availability of machine-readable research data has created a growing need for efficient large-scale data harmonization (DH). Although large language models (LLMs) show promise for reducing the time and labor required for DH, their effective integration into harmonization workflows remains an important methodological challenge.Objective
This study aims to develop and evaluate a semiautomated, human-in-the-loop (HITL) LLM-assisted DH workflow as a proof of concept across 8 projects within a research consortium focused on disparities in multiple chronic conditions.Methods
We developed an LLM-assisted DH workflow that combined preprocessing, semantic mapping, variable response mapping, synthetic data generation, data transformation, and iterative researcher review. Using Generative Pretrained Transformer (GPT-4o) hosted in a secure academic environment, project-specific survey items were mapped to 114 Consortium Common Data Element (CDE) semantic groups. Semantic mapping accuracy was evaluated based on the Consortium's consensus harmonization results. Complete and incomplete semantic mappings were distinguished according to construct overlap, and incomplete mappings requiring context-dependent human judgment were excluded for mapped item pairs and CDE semantic groups with and without HITL review.Results
Following preprocessing, 884 survey items were included in the DH workflow, of which 812 were mapped to a mean of 79 CDE semantic groups per project. Mean semantic mapping accuracy for item pairs was 95.9% (SD 5.7%) with HITL review and 86.7% (SD 8.6%) without HITL review. At the CDE semantic-group level, the corresponding accuracies were 98.9% (SD 1.7%) and 94.9% (SD 4.0%), respectively. Across projects, omission of valid semantic mappings occurred more frequently than incorrect semantic mappings, particularly for heterogeneous response structures and subjective constructs measured using different instruments. Pipeline components-including concept-based preprocessing, iterative prompt refinement, and targeted HITL review-improved mapping completeness and supported variable harmonization across heterogeneous datasets.Conclusions
This proof-of-concept study demonstrates that the effectiveness of LLM-assisted DH depends more on the design of a structured HITL workflow rather than on the LLM alone. Combining preprocessing, iterative verification, and targeted human oversight improved the efficiency and reliability of DH while preserving researcher judgment for complex mapping decisions.
Related
- Retrospective harmonization of neuropsychiatric questionnaires in a large multi-site research consortium using expert mappings and semantic textual similarity
- The Past and Future of Fungal Diagnoses in Health Records: A Comparative Analysis and Harmonization of Mycoses ICD Codes
- AAV serotype IVDs for patient selection in gene therapy development: Technical and regulatory considerations by ARDAT consortium
- Semiautomated Synthesis and Interactive Visualization of Zeolitic Imidazolate Framework-Based Biocomposites
- LASSBio-classFLOW: A Semiautomated KNIME Workflow for Classification Model Benchmarking and Virtual Screening
BibTeX
@article{kim2026large,
title = {{Large Language Model-Assisted Semiautomated Data Harmonization Pipeline for the Multiple Chronic Disease Disparities Research Consortium: Framework Development and Comparative Validation Study.}},
author = {Hyelee Kim and Shuang Liang and Kathy Lanier and Sarit Helman and Nancy F Cheng and Niloufar Ameli and Jaysón Davidson and Shivani Mehta and Yulin Hswen and Catherine E Oldenburg and Kim F Rhoads and Edwin D Charlebois and Stuart A Gansky and William Brown Iii},
journal = {JMIR AI},
year = {2026},
month = oct,
doi = {10.2196/93996},
url = {https://europepmc.org/article/MED/42912962},
note = {CC BY},
}
RIS
TY - JOUR
TI - Large Language Model-Assisted Semiautomated Data Harmonization Pipeline for the Multiple Chronic Disease Disparities Research Consortium: Framework Development and Comparative Validation Study.
AU - Hyelee Kim
AU - Shuang Liang
AU - Kathy Lanier
AU - Sarit Helman
AU - Nancy F Cheng
AU - Niloufar Ameli
AU - Jaysón Davidson
AU - Shivani Mehta
AU - Yulin Hswen
AU - Catherine E Oldenburg
AU - Kim F Rhoads
AU - Edwin D Charlebois
AU - Stuart A Gansky
AU - William Brown Iii
PY - 2026
DA - 2026/10/22
JO - JMIR AI
DO - 10.2196/93996
UR - https://europepmc.org/article/MED/42912962
AB - Background The increasing availability of machine-readable research data has created a growing need for efficient large-scale data harmonization (DH). Although large language models (LLMs) show promise for reducing the time and labor required for DH, their effective integration into harmonization workflows remains an important methodological challenge.Objective This study aims to develop and evaluate a semiautomated, human-in-the-loop (HITL) LLM-assisted DH workflow as a proof of concept across 8 projects within a research consortium focused on disparities in multiple chronic conditions.Methods We developed an LLM-assisted DH workflow that combined preprocessing, semantic mapping, variable response mapping, synthetic data generation, data transformation, and iterative researcher review. Using Generative Pretrained Transformer (GPT-4o) hosted in a secure academic environment, project-specific survey items were mapped to 114 Consortium Common Data Element (CDE) semantic groups. Semantic mapping accuracy was evaluated based on the Consortium's consensus harmonization results. Complete and incomplete semantic mappings were distinguished according to construct overlap, and incomplete mappings requiring context-dependent human judgment were excluded for mapped item pairs and CDE semantic groups with and without HITL review.Results Following preprocessing, 884 survey items were included in the DH workflow, of which 812 were mapped to a mean of 79 CDE semantic groups per project. Mean semantic mapping accuracy for item pairs was 95.9% (SD 5.7%) with HITL review and 86.7% (SD 8.6%) without HITL review. At the CDE semantic-group level, the corresponding accuracies were 98.9% (SD 1.7%) and 94.9% (SD 4.0%), respectively. Across projects, omission of valid semantic mappings occurred more frequently than incorrect semantic mappings, particularly for heterogeneous response structures and subjective constructs measured using different instruments. Pipeline components-including concept-based preprocessing, iterative prompt refinement, and targeted HITL review-improved mapping completeness and supported variable harmonization across heterogeneous datasets.Conclusions This proof-of-concept study demonstrates that the effectiveness of LLM-assisted DH depends more on the design of a structured HITL workflow rather than on the LLM alone. Combining preprocessing, iterative verification, and targeted human oversight improved the efficiency and reliability of DH while preserving researcher judgment for complex mapping decisions.
SN - 2817-1705
ER -