Design and Technical Evaluation of a Standards-Based System That Delivers Enclave-Derived Case-Finding Models to Patients and Clinicians
- 1Vanderbilt University Medical Center
Abstract
BackgroundResearch data enclaves such as the All of Us Research Program do not allow participant-level data, or models fitted to them, to leave, which makes it hard to deliver case-finding models, which flag patients with possibly uncoded conditions for clinician review, to patients and clinicians.
ObjectivesWe describe and technically evaluate a system that fits models outside an enclave from released summary counts and delivers them through standard health information technology interfaces.
MethodsWe exported dissemination-compliant summary counts from the All of Us Researcher Workbench and, outside the enclave, fitted versioned models for Long COVID, chronic fatigue, fibromyalgia, endometriosis, Lyme disease, lupus, migraine, depression, and anxiety. A model contract (a model file and a written specification), a Python reference implementation, and synthetic Fast Healthcare Interoperability Resources (FHIR) test patients define the expected outputs. One JavaScript scorer runs in a patient demonstration, a SMART on FHIR clinician application, and a CDS Hooks service. An atlas displays aggregate results. We evaluated the system on synthetic records with automated tests, cross-implementation output comparison, launches through the public SMART sandbox, and a counterexample review of the scoring and display rules.
ResultsThe deployed Long COVID model had an area under the receiver operating characteristic curve of 0.67 in a random validation split. The eight self-report models reached 0.61 to 0.82 in identifying self-reporters without a code. The JavaScript scorer reproduced all 162 reference outputs exactly, and all 437 automated tests passed. A counterexample review found 5 classes of failure, including scoring ineligible patients and treating unavailable data as absent, all fixed and added as tests.
ConclusionsModels fitted from released summary counts can be delivered through FHIR, SMART on FHIR, and CDS Hooks with reproducible scoring on synthetic records. Real-record interoperability, usability, and prospective clinical performance remain to be evaluated.
Related
- Evaluation of Clinicians’ Knowledge of Terbinafine-resistant Extensive Dermatophytosis in France
- Toward Standards for Responsible AI in Health Communication: A Dual Method Pre-Deployment Risk Evaluation Framework and Case Study
- Perioperative analgesia in high-risk patients, what clinicians and searchers must know? A narrative review
- Predicting the Risk of Unplanned Hospital Encounters in Hemodialysis Patients: A Comparison Between Clinicians and a Machine Learning Prediction Algorithm
- A systematic review of ECG arrhythmia classification: embedded feasibility, adherence to standards, and fair evaluation
BibTeX
@misc{asiaee2026design,
title = {{Design and Technical Evaluation of a Standards-Based System That Delivers Enclave-Derived Case-Finding Models to Patients and Clinicians}},
author = {A. Asiaee and K. Aryan},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.20.26365982},
url = {https://www.medrxiv.org/content/10.64898/2026.10.20.26365982v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Design and Technical Evaluation of a Standards-Based System That Delivers Enclave-Derived Case-Finding Models to Patients and Clinicians
AU - A. Asiaee
AU - K. Aryan
PY - 2026
DA - 2026/10/20
PB - medRxiv
DO - 10.64898/2026.10.20.26365982
UR - https://www.medrxiv.org/content/10.64898/2026.10.20.26365982v1
AB - BackgroundResearch data enclaves such as the All of Us Research Program do not allow participant-level data, or models fitted to them, to leave, which makes it hard to deliver case-finding models, which flag patients with possibly uncoded conditions for clinician review, to patients and clinicians. ObjectivesWe describe and technically evaluate a system that fits models outside an enclave from released summary counts and delivers them through standard health information technology interfaces. MethodsWe exported dissemination-compliant summary counts from the All of Us Researcher Workbench and, outside the enclave, fitted versioned models for Long COVID, chronic fatigue, fibromyalgia, endometriosis, Lyme disease, lupus, migraine, depression, and anxiety. A model contract (a model file and a written specification), a Python reference implementation, and synthetic Fast Healthcare Interoperability Resources (FHIR) test patients define the expected outputs. One JavaScript scorer runs in a patient demonstration, a SMART on FHIR clinician application, and a CDS Hooks service. An atlas displays aggregate results. We evaluated the system on synthetic records with automated tests, cross-implementation output comparison, launches through the public SMART sandbox, and a counterexample review of the scoring and display rules. ResultsThe deployed Long COVID model had an area under the receiver operating characteristic curve of 0.67 in a random validation split. The eight self-report models reached 0.61 to 0.82 in identifying self-reporters without a code. The JavaScript scorer reproduced all 162 reference outputs exactly, and all 437 automated tests passed. A counterexample review found 5 classes of failure, including scoring ineligible patients and treating unavailable data as absent, all fixed and added as tests. ConclusionsModels fitted from released summary counts can be delivered through FHIR, SMART on FHIR, and CDS Hooks with reproducible scoring on synthetic records. Real-record interoperability, usability, and prospective clinical performance remain to be evaluated.
ER -