Forecasting accuracy is not enough: safety-oriented evaluation of neural digital twins prior to in-silico insulin and carbohydrate intervention in type 1 diabetes
5 affiliations
- 1Department of Computer Science and Artificial Intelligence, College of Computer Science and Engineering, University of Jeddah
- 2Department of Information Systems and Technology, College of Computer Science and Engineering, University of Jeddah
- 3Department of Computer Science, The University of Manchester
- 4Department of Information Technology, College of Computer Science and Engineering, University of Jeddah
- 5Department of Cybersecurity, College of Computer Science and Engineering, University of Jeddah
Abstract
Neural networks trained on a single patient’s continuous glucose monitoring (CGM) record are increasingly proposed as “digital twins” that could let clinicians rehearse insulin and carbohydrate decisions in silico before applying them to the patient. Before any such use, a candidate twin should respond to treatment in a direction consistent with established pharmacology: a model that merely extrapolates the recent glucose trend can report an excellent forecasting error while remaining effectively blind to insulin, and this failure is invisible to the root-mean-square error (RMSE) that dominates the glucose-forecasting literature. We examined this gap in two independent cohorts: a longitudinal, multimodal University of Manchester cohort (17 participants enrolled; 13 formed the Core analytical cohort after data-quality screening) and, as an independent replication of the protocol, the OhioT1DM dataset (12 participants). Patient-specific two-layer long short-term memory (LSTM) models were trained under a strict chronological 60/20/20 split and evaluated across five random seeds. A physiology-guided variant added insulin-on-board and carbohydrate-on-board states with a double-monotonicity training penalty encouraging insulin to lower and carbohydrate to raise predicted glucose. Two findings replicated across both cohorts and all seeds. First, on the accuracy axis, a physiology-free Vanilla LSTM had the lowest one-step RMSE (Manchester 7.00, Ohio 5.02 mg/dL; Friedman χ ² (5) = 43.4 and 46.7, both p < 10 ⁻ ⁷ ), and under two-hour rollout a non-learning persistence baseline beat every neural model at 60 and 120 minutes—even after rollout-aware training—so ranking twins by accuracy alone would select a model that cannot represent treatment at all. Second, on a graded insulin dose-response audit, the physiology-guided model produced the steepest correctly-directed (glucose-lowering) response among the primary configurations in both cohorts (slope significantly better than the unconstrained model: Holm-corrected p = 0.002 and 0.003), whereas removing the directional constraint left a wrong-signed response; a supplementary ablation without masking was steeper still, pointing to the directional penalty, not masking, as the driver. Its advantage over the simpler Vanilla model was significant on for this cohort dependence. Accuracy and treatment-response plausibility are separable properties; we provide a reproducible protocol auditing both before a neural twin is used for intervention simulation.
Related
- Effect of In-line Clinical Decision Support to Improve Accuracy of American Society of Anesthesiologists Physical Status Documentation: A Two Center Evaluation of a Quality Intervention
- Beyond overall accuracy: subgroup performance and threshold calibration of AI-assisted chest radiography for equity-oriented pulmonary tuberculosis screening.
- Mechanisms of Benefit in Geriatric Assessment (GA): Approach‐Oriented Coping Mediates Outcomes of a GA‐Guided Supportive Care Intervention (GAIN‐S)
- Effects of a Web-Based Debiasing Intervention on Brazilian Psychologists’ Accuracy on Major Depressive Disorder Diagnostic Tasks: A Randomized Training Intervention
- Effectiveness of a brief body-oriented intervention on muscle activities when playing the violin.
BibTeX
@article{ibrahim2026forecasting,
title = {{Forecasting accuracy is not enough: safety-oriented evaluation of neural digital twins prior to in-silico insulin and carbohydrate intervention in type 1 diabetes}},
author = {Ahmed Ibrahim and Ameen Banjar and Simon Harper and Ala Alarood and Eesa Alsolami},
journal = {Frontiers in Endocrinology},
year = {2026},
month = oct,
doi = {10.3389/fendo.2026.1958918},
url = {https://www.frontiersin.org/articles/10.3389/fendo.2026.1958918/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Forecasting accuracy is not enough: safety-oriented evaluation of neural digital twins prior to in-silico insulin and carbohydrate intervention in type 1 diabetes
AU - Ahmed Ibrahim
AU - Ameen Banjar
AU - Simon Harper
AU - Ala Alarood
AU - Eesa Alsolami
PY - 2026
DA - 2026/10/14
JO - Frontiers in Endocrinology
DO - 10.3389/fendo.2026.1958918
UR - https://www.frontiersin.org/articles/10.3389/fendo.2026.1958918/full
AB - Neural networks trained on a single patient’s continuous glucose monitoring (CGM) record are increasingly proposed as “digital twins” that could let clinicians rehearse insulin and carbohydrate decisions in silico before applying them to the patient. Before any such use, a candidate twin should respond to treatment in a direction consistent with established pharmacology: a model that merely extrapolates the recent glucose trend can report an excellent forecasting error while remaining effectively blind to insulin, and this failure is invisible to the root-mean-square error (RMSE) that dominates the glucose-forecasting literature. We examined this gap in two independent cohorts: a longitudinal, multimodal University of Manchester cohort (17 participants enrolled; 13 formed the Core analytical cohort after data-quality screening) and, as an independent replication of the protocol, the OhioT1DM dataset (12 participants). Patient-specific two-layer long short-term memory (LSTM) models were trained under a strict chronological 60/20/20 split and evaluated across five random seeds. A physiology-guided variant added insulin-on-board and carbohydrate-on-board states with a double-monotonicity training penalty encouraging insulin to lower and carbohydrate to raise predicted glucose. Two findings replicated across both cohorts and all seeds. First, on the accuracy axis, a physiology-free Vanilla LSTM had the lowest one-step RMSE (Manchester 7.00, Ohio 5.02 mg/dL; Friedman χ ² (5) = 43.4 and 46.7, both p < 10 ⁻ ⁷ ), and under two-hour rollout a non-learning persistence baseline beat every neural model at 60 and 120 minutes—even after rollout-aware training—so ranking twins by accuracy alone would select a model that cannot represent treatment at all. Second, on a graded insulin dose-response audit, the physiology-guided model produced the steepest correctly-directed (glucose-lowering) response among the primary configurations in both cohorts (slope significantly better than the unconstrained model: Holm-corrected p = 0.002 and 0.003), whereas removing the directional constraint left a wrong-signed response; a supplementary ablation without masking was steeper still, pointing to the directional penalty, not masking, as the driver. Its advantage over the simpler Vanilla model was significant on for this cohort dependence. Accuracy and treatment-response plausibility are separable properties; we provide a reproducible protocol auditing both before a neural twin is used for intervention simulation.
SN - 1664-2392
ER -