Workflow Topology–Guided Skill Loading for Agentic AI in Clinical Trial Data Pipelines: Sequential Exploratory Evaluation
- 1Department of Data Science Harrisburg University of Science and Technology Harrisburg, PA United States
Abstract
Background: Clinical trial data pipelines must follow Clinical Data Interchange Standards Consortium (CDISC) standards, including the study data tabulation model (SDTM) and analysis data model (ADaM). General-purpose large language models (LLMs) can produce plausible artifacts that omit required variables, misuse terminology, or invent derivation logic. Curated context may reduce these errors, but the value of organizing that context by workflow topology has not been separated from corpus access alone.
Objective: This study aimed to examine when workflow topology–guided skill loading adds value beyond a domain prompt, nonrouted curated context, or lexical retrieval, and whether bounded scheduling retains quality with less injected context.
Methods: A 45-node, 7-layer directed acyclic workflow graph was linked to 59 modular rule files (SKILL.md). A sequential exploratory evaluation examined 4 evidence regimes spanning domain-context, component isolation, lexically reachable, and topology-dependent conditions (10 tasks, 90 runs; 10 tasks, 180 and 120 runs; 24 tasks, 288 runs; and 18 tasks, 324 runs; the task, not the agent run, was the inferential unit). End points included LLM rubric scores, deterministic atomic criteria, required-evidence recall, and manifest checks. A blinded practicing statistical programmer independently rescored 60 outputs. Task-level paired inference used bootstrap 95% CIs, exact sign-flip tests, and Holm adjustment.
Results: Routed context scored 4.63 versus 4.20 for prompt-only (difference 0.43 on a 1-5 scale; 95% CI 0.25-0.63; Holm-adjusted P=.01) but did not exceed flat or random loading. The lower-context scheduler used 28.82% less context while retaining quality; all 559 injected fragments had recorded sources, and all 120 manifests were reproduced exactly. On lexically reachable tasks, graph+Best Match 25 (BM25) and BM25-only were equivalent within a 0.05 margin (difference +0.009; 95% CI –0.028 to 0.046; P=.67), where BM25 already recalled 95.8% of required evidence. On topology-dependent stress tasks, graph-only selection outperformed BM25-only (difference 0.348; 95% CI 0.178-0.530; nominal P=.002; dz=0.89) and raised required-evidence recall from 0.500 to 1.000. Expert-judge agreement was substantial for terminology (weighted κ=0.61) and moderate overall (intraclass correlation coefficient=0.60).
Conclusions: Workflow topology serves as a conditional retrieval prior. It adds little when lexical retrieval already reaches the required evidence but recovers dependency-linked evidence when task wording does not identify it lexically. These findings, from a synthetic benchmark using one generator backbone and one expert rater, support topology-guided selection as a promising approach for controlled context assembly and motivate independent factorial validation.
Related
- Development and psychometric properties evaluation of individualized patient education tool: An exploratory sequential mixed-methods study protocol
- An Optimized Freeze‐Dry Multimodal Workflow for Sequential Micro‐CT Imaging, Histology, and Molecular Profiling: A Use Case in Human Liver Fibrosis
- Perceived factors influencing healthcare professionals’ adherence to the ABCDE approach: an exploratory explanatory sequential mixed-methods study
- Same pipelines, different answers: how evaluation design shapes EEG seizure prediction
- Exploring stakeholder support for student-run free clinics to address gaps in non-communicable disease care in Ghana: An exploratory sequential mixed-methods study
BibTeX
@article{yan2026workflow,
title = {{Workflow Topology–Guided Skill Loading for Agentic AI in Clinical Trial Data Pipelines: Sequential Exploratory Evaluation}},
author = {Jaime Yan},
journal = {JMIR Formative Research},
year = {2026},
month = oct,
doi = {10.2196/100769},
url = {https://doi.org/10.2196/100769},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Workflow Topology–Guided Skill Loading for Agentic AI in Clinical Trial Data Pipelines: Sequential Exploratory Evaluation
AU - Jaime Yan
PY - 2026
DA - 2026/10/20
JO - JMIR Formative Research
DO - 10.2196/100769
UR - https://doi.org/10.2196/100769
AB - Background: Clinical trial data pipelines must follow Clinical Data Interchange Standards Consortium (CDISC) standards, including the study data tabulation model (SDTM) and analysis data model (ADaM). General-purpose large language models (LLMs) can produce plausible artifacts that omit required variables, misuse terminology, or invent derivation logic. Curated context may reduce these errors, but the value of organizing that context by workflow topology has not been separated from corpus access alone. Objective: This study aimed to examine when workflow topology–guided skill loading adds value beyond a domain prompt, nonrouted curated context, or lexical retrieval, and whether bounded scheduling retains quality with less injected context. Methods: A 45-node, 7-layer directed acyclic workflow graph was linked to 59 modular rule files (SKILL.md). A sequential exploratory evaluation examined 4 evidence regimes spanning domain-context, component isolation, lexically reachable, and topology-dependent conditions (10 tasks, 90 runs; 10 tasks, 180 and 120 runs; 24 tasks, 288 runs; and 18 tasks, 324 runs; the task, not the agent run, was the inferential unit). End points included LLM rubric scores, deterministic atomic criteria, required-evidence recall, and manifest checks. A blinded practicing statistical programmer independently rescored 60 outputs. Task-level paired inference used bootstrap 95% CIs, exact sign-flip tests, and Holm adjustment. Results: Routed context scored 4.63 versus 4.20 for prompt-only (difference 0.43 on a 1-5 scale; 95% CI 0.25-0.63; Holm-adjusted P=.01) but did not exceed flat or random loading. The lower-context scheduler used 28.82% less context while retaining quality; all 559 injected fragments had recorded sources, and all 120 manifests were reproduced exactly. On lexically reachable tasks, graph+Best Match 25 (BM25) and BM25-only were equivalent within a 0.05 margin (difference +0.009; 95% CI –0.028 to 0.046; P=.67), where BM25 already recalled 95.8% of required evidence. On topology-dependent stress tasks, graph-only selection outperformed BM25-only (difference 0.348; 95% CI 0.178-0.530; nominal P=.002; dz=0.89) and raised required-evidence recall from 0.500 to 1.000. Expert-judge agreement was substantial for terminology (weighted κ=0.61) and moderate overall (intraclass correlation coefficient=0.60). Conclusions: Workflow topology serves as a conditional retrieval prior. It adds little when lexical retrieval already reaches the required evidence but recovers dependency-linked evidence when task wording does not identify it lexically. These findings, from a synthetic benchmark using one generator backbone and one expert rater, support topology-guided selection as a promising approach for controlled context assembly and motivate independent factorial validation.
SN - 2561-326X
ER -