External validation and adaptation of the Computational Barthel Index Tool in a large nursing home cohort
- 1Department of Health Administration, Policy and Informatics, George Mason University, Fairfax, Virginia, United States of America.
Abstract
The Computational Barthel Index Tool (CBIT), originally developed using U.S. Department of Veterans Affairs (VA) data, predicts future dependency in activities of daily living up to one year in advance. This study externally validated CBIT in a large non-VA nursing home population and evaluated whether population-specific adaptation could recover performance following external transport. We analyzed SEER-Medicare linked data, including 843,836 patients and over 8 million Minimum Data Set (MDS) assessments. The predictor and outcome structures of the original CBIT models were reconstructed in the SEER-Medicare cohort. Pretrained models were evaluated without modification for discrimination and calibration at current, 3-, 6-, and 12-month prediction timeframes. Population-specific adaptation was evaluated using progressively larger subsets of SEER-Medicare training data and compared with model reconstruction in the SEER-Medicare. Under strict external validation, discrimination declined relative to the original VA models, particularly for longer-term predictions. For evaluation models, mean AUC decreased from 0.795 in the original models to 0.693 at current prediction, 0.595 at 3 months, 0.595 at 6 months, and 0.592 at 12 months. Re-evaluation models showed stronger but similarly declining external performance. Calibration error increased, particularly for evaluation models at longer timeframes. Subgroup analyses showed modest variation in discrimination across demographic and clinical groups. Population-specific adaptation substantially recovered discrimination, with performance improving as more SEER-Medicare training data were incorporated. Using the adaptation training set, adapted models achieved performance comparable to SEER-Medicare model reconstruction. External validation revealed substantial performance degradation when CBIT was transported to a non-VA population. Population-specific adaptation recovered much of this performance loss, supporting a structured approach to clinical ML transportability in which pretrained models are externally validated first, followed by adaptation when performance degradation is observed. Full model reconstruction remains an alternative when sufficient local data are available. Author summary: Machine learning models are increasingly used in healthcare to help predict patient outcomes and support clinical decision-making. However, models developed in one healthcare system may not perform as well when applied to different patient populations. Understanding how performance changes when a model is evaluated in a new population, and whether performance can be recovered without fully redeveloping the model, is important for successful implementation of clinical machine learning. We evaluated the Computational Barthel Index Tool (CBIT), a model developed using U.S. Department of Veterans Affairs (VA) data to predict future dependency in activities of daily living. We tested the model in a non-VA nursing home population of more than 800,000 patients, including millions of clinical assessments. When applied without modification, CBIT performance declined, particularly for longer-term predictions. We then adapted the model using data from the new population and examined how performance changed as increasing amounts of local data were used. Adaptation substantially recovered performance, with greater improvements observed as more local data were incorporated. We also compared adapted models with models reconstructed using local data. These findings suggest that adapting an existing model may provide a practical alternative to full model redevelopment when local data or development resources are limited.
Related
- Real-World Adaptation and External Validation of Machine-Learning Impedance Spectroscopy for Immediate Postpartum Detection of Obstetric Anal Sphincter Injuries
- Cross-cultural adaptation and psychometric evaluation of the Chinese version of the Digital and Technological Self-Efficacy Scale among nursing students: a three-phase validation study.
- Undergraduate nursing students' clinical adaptation: a qualitative journey mapping study from student and clinical preceptor perspectives
- External validation of the eurolung risk model for morbidity using an institutional database on minimally invasive major lung resections
- Dynamic serum chloride trajectories predict short- and long-term mortality in patients with malignancies: a retrospective cohort study with external validation
BibTeX
@article{ngana2026external,
title = {{External validation and adaptation of the Computational Barthel Index Tool in a large nursing home cohort}},
author = {Lemba Priscille Ngana and Janusz Wojtusiak and Atefehsadat Haghighathoseini and Bhumi Patel and Dharmi Desai},
journal = {PLOS Digital Health},
year = {2026},
month = oct,
doi = {10.1371/journal.pdig.0001793},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001793},
note = {CC BY},
}
RIS
TY - JOUR
TI - External validation and adaptation of the Computational Barthel Index Tool in a large nursing home cohort
AU - Lemba Priscille Ngana
AU - Janusz Wojtusiak
AU - Atefehsadat Haghighathoseini
AU - Bhumi Patel
AU - Dharmi Desai
PY - 2026
DA - 2026/10/23
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001793
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001793
AB - The Computational Barthel Index Tool (CBIT), originally developed using U.S. Department of Veterans Affairs (VA) data, predicts future dependency in activities of daily living up to one year in advance. This study externally validated CBIT in a large non-VA nursing home population and evaluated whether population-specific adaptation could recover performance following external transport. We analyzed SEER-Medicare linked data, including 843,836 patients and over 8 million Minimum Data Set (MDS) assessments. The predictor and outcome structures of the original CBIT models were reconstructed in the SEER-Medicare cohort. Pretrained models were evaluated without modification for discrimination and calibration at current, 3-, 6-, and 12-month prediction timeframes. Population-specific adaptation was evaluated using progressively larger subsets of SEER-Medicare training data and compared with model reconstruction in the SEER-Medicare. Under strict external validation, discrimination declined relative to the original VA models, particularly for longer-term predictions. For evaluation models, mean AUC decreased from 0.795 in the original models to 0.693 at current prediction, 0.595 at 3 months, 0.595 at 6 months, and 0.592 at 12 months. Re-evaluation models showed stronger but similarly declining external performance. Calibration error increased, particularly for evaluation models at longer timeframes. Subgroup analyses showed modest variation in discrimination across demographic and clinical groups. Population-specific adaptation substantially recovered discrimination, with performance improving as more SEER-Medicare training data were incorporated. Using the adaptation training set, adapted models achieved performance comparable to SEER-Medicare model reconstruction. External validation revealed substantial performance degradation when CBIT was transported to a non-VA population. Population-specific adaptation recovered much of this performance loss, supporting a structured approach to clinical ML transportability in which pretrained models are externally validated first, followed by adaptation when performance degradation is observed. Full model reconstruction remains an alternative when sufficient local data are available. Author summary: Machine learning models are increasingly used in healthcare to help predict patient outcomes and support clinical decision-making. However, models developed in one healthcare system may not perform as well when applied to different patient populations. Understanding how performance changes when a model is evaluated in a new population, and whether performance can be recovered without fully redeveloping the model, is important for successful implementation of clinical machine learning. We evaluated the Computational Barthel Index Tool (CBIT), a model developed using U.S. Department of Veterans Affairs (VA) data to predict future dependency in activities of daily living. We tested the model in a non-VA nursing home population of more than 800,000 patients, including millions of clinical assessments. When applied without modification, CBIT performance declined, particularly for longer-term predictions. We then adapted the model using data from the new population and examined how performance changed as increasing amounts of local data were used. Adaptation substantially recovered performance, with greater improvements observed as more local data were incorporated. We also compared adapted models with models reconstructed using local data. These findings suggest that adapting an existing model may provide a practical alternative to full model redevelopment when local data or development resources are limited.
SN - 2767-3170
ER -