Patient-level consequences of local updates to retinal AI for diabetic retinopathy screening: a retrospective two-cohort evaluation
- 1School of Biomedical Engineering, Capital Medical University
Abstract
ObjectivesTo assess whether local artificial intelligence updates for diabetic retinopathy screening that improve image-level probabilities also improve patient-level detection, flagging and net benefit.
MethodsThis retrospective evaluation used four systems, BRSET source data and two external cohorts, mBRSET and DeepDRiD, with separate audit and evaluation patients. Updates were fitted and selected on audits of 25-200 patients; probability recalibration and operating policies were selected separately. Outcomes included Brier loss, patient detection, flags and net benefit at four threshold probabilities (p=0.05-0.30). Comparators were no update, flag all and flag none. Conditional patient-cluster intervals compared updates with no update; simple-policy comparisons were descriptive.
ResultsAt audit budget 200, mean Brier loss improved in all eight cohort-system comparisons. In mBRSET, flags rose from 154-442 to 809-951 per 1,000 patients and missed positives fell from 51-133 to 2-14 (5.69-9.48 additional flags per additional detection). In mBRSET at p=0.05, three of four updates had lower mean net benefit than flag all; at p=0.20, all four exceeded flag all but remained below no update. In DeepDRiD at p=0.20, the APTOS update increased mean net benefit by 0.0510 over no update and exceeded both simple policies.
DiscussionImage-level improvement did not ensure patient-level benefit. Flag increases were consistent with image-level targets applied to patients with about four images.
ConclusionLocal updates should be judged on patient-level detection, flags and net benefit against unchanged, flag-all and flag-none policies, not on image-level metrics alone.
KEY MESSAGESO_ST_ABSWhat is already known on this topicC_ST_ABSO_LIRetinal AI performance can change between settings, and local recalibration or threshold adjustment can improve conventional evaluation metrics. C_LI
What this study addsO_LILocal updates improved image-level Brier loss in all eight cohort-system comparisons, but in mBRSET at p=0.20 all four lowered mean patient-level net benefit, adding 5.7-9.5 positive flags per additional detection. Whether an update outperformed the unchanged, flag-all and flag-none policies depended on the cohort and threshold probability. C_LI
How this study might affect research, practice or policyO_LIBefore adopting a local update, services can compare patient detection, positive flags and net benefit with the unchanged, flag-all and flag-none policies on held-out patients. C_LI
Related
- Performance Assessment of an Artificial Intelligence System Used for Case Detection in a Large Community-based Diabetic Retinopathy Screening Program: A Retrospective Diagnostic Accuracy Validation Study.
- A Robust and Trustable Approach to Incorporate Medical Domain Knowledge in Machine Learning Models for Diabetic Retinopathy Screening Using Routine Lab Results
- ACCESS2 study: the potential for autonomous AI to mitigate disparities in diabetic retinopathy screening in youth
- Artificial intelligence-supported diabetic retinopathy screening in Tanzania: a randomised controlled trial
- Clinical setting-dependent diagnostic accuracy of artificial intelligence and store-and-forward diabetic retinopathy screening: a systematic review and meta-analysis
BibTeX
@misc{li2026patientlevel,
title = {{Patient-level consequences of local updates to retinal AI for diabetic retinopathy screening: a retrospective two-cohort evaluation}},
author = {R. Li and T. Ma and L. Tang and S. Li and Y. Wu and H. Xia},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.23.26366103},
url = {https://www.medrxiv.org/content/10.64898/2026.10.23.26366103v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Patient-level consequences of local updates to retinal AI for diabetic retinopathy screening: a retrospective two-cohort evaluation
AU - R. Li
AU - T. Ma
AU - L. Tang
AU - S. Li
AU - Y. Wu
AU - H. Xia
PY - 2026
DA - 2026/10/23
PB - medRxiv
DO - 10.64898/2026.10.23.26366103
UR - https://www.medrxiv.org/content/10.64898/2026.10.23.26366103v1
AB - ObjectivesTo assess whether local artificial intelligence updates for diabetic retinopathy screening that improve image-level probabilities also improve patient-level detection, flagging and net benefit. MethodsThis retrospective evaluation used four systems, BRSET source data and two external cohorts, mBRSET and DeepDRiD, with separate audit and evaluation patients. Updates were fitted and selected on audits of 25-200 patients; probability recalibration and operating policies were selected separately. Outcomes included Brier loss, patient detection, flags and net benefit at four threshold probabilities (p=0.05-0.30). Comparators were no update, flag all and flag none. Conditional patient-cluster intervals compared updates with no update; simple-policy comparisons were descriptive. ResultsAt audit budget 200, mean Brier loss improved in all eight cohort-system comparisons. In mBRSET, flags rose from 154-442 to 809-951 per 1,000 patients and missed positives fell from 51-133 to 2-14 (5.69-9.48 additional flags per additional detection). In mBRSET at p=0.05, three of four updates had lower mean net benefit than flag all; at p=0.20, all four exceeded flag all but remained below no update. In DeepDRiD at p=0.20, the APTOS update increased mean net benefit by 0.0510 over no update and exceeded both simple policies. DiscussionImage-level improvement did not ensure patient-level benefit. Flag increases were consistent with image-level targets applied to patients with about four images. ConclusionLocal updates should be judged on patient-level detection, flags and net benefit against unchanged, flag-all and flag-none policies, not on image-level metrics alone. KEY MESSAGESO_ST_ABSWhat is already known on this topicC_ST_ABSO_LIRetinal AI performance can change between settings, and local recalibration or threshold adjustment can improve conventional evaluation metrics. C_LI What this study addsO_LILocal updates improved image-level Brier loss in all eight cohort-system comparisons, but in mBRSET at p=0.20 all four lowered mean patient-level net benefit, adding 5.7-9.5 positive flags per additional detection. Whether an update outperformed the unchanged, flag-all and flag-none policies depended on the cohort and threshold probability. C_LI How this study might affect research, practice or policyO_LIBefore adopting a local update, services can compare patient detection, positive flags and net benefit with the unchanged, flag-all and flag-none policies on held-out patients. C_LI
ER -