Preserving a decade of machine-learning validity across a ground-up refactoring: a five-tier automated validation of RCT-Reviewer, an independent modernization of RobotReviewer
- 1Georgian National University SEU
Abstract
Evidence-synthesis teams increasingly depend on machine-learning tools to automate risk-of-bias assessment, but these tools frequently rely on unrunnable, deprecated software stacks. Refactoring legacy tools into modern environments is essential for maintenance, yet introduces a critical risk: silently invalidating published performance metrics. Without rigorous validation, systematic reviewers cannot trust that modernized tools retain their predecessors behavior. We present a five-tier validation framework applied to RCT-Reviewer, an independent ground-up modernization of the widely used RobotReviewer system. To prove preservation, we engineered a compatibility shim to execute the original 2017 code alongside the modernized tool using byte-identical weight files. The framework evaluated predictive validity on a human-labelled benchmark, quantified the cost of dropping obsolete ensemble components, proved bit-exact fidelity, tested parser robustness at scale, and included an external validation arm evaluating agreement on 313 trials against a recent reference standard. A control experiment showing 100% agreement between legacy and modernized implementations on identical text proved that variance versus published values is driven by PDF provenance (open-access versus publisher typeset) rather than codebase refactoring. The modernized tool perfectly reproduced original risk-of-bias judgements in 6,018 comparisons. While the screening classifier experienced a bounded performance reduction (F1 0.969 to 0.925) from removing obsolete components, the models core mathematical validity is preserved. These results establish that reviewers can deploy a maintainable modernization with a clearly bounded performance trade-off, offering a reusable validation framework for clinical NLP modernizations. Future work will explore retraining obsolete neural network components to close the performance gap.
Related
- Federated learning for privacy-preserving opportunistic screening of lumbar-spine DXA-defined osteoporotic-level bone mineral density using chest radiographs: A multi-center independent validation study
- NRF2 signalling in endometriosis: A hormone-independent axis for fertility-preserving therapeutic innovation
- An automated operon refactoring workflow for heterologous expression of non-ribosomal peptide biosynthetic gene clusters
- Transcriptomic Profiling Reveals METTL26 as a Potential Biomarker in Head and Neck Squamous Cell Carcinoma with Independent Validation in Oral Squamous Cell Carcinoma
- Serum Free Fatty Acid-Based Diagnostic Index for Cervical Cancer Including High-Risk HPV-Negative Cases: An Independent Validation Study
BibTeX
@misc{sahu2026preserving,
title = {{Preserving a decade of machine-learning validity across a ground-up refactoring: a five-tier automated validation of RCT-Reviewer, an independent modernization of RobotReviewer}},
author = {V. Sahu and A. Sahu},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.08.26362546},
url = {https://www.medrxiv.org/content/10.64898/2026.09.08.26362546v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Preserving a decade of machine-learning validity across a ground-up refactoring: a five-tier automated validation of RCT-Reviewer, an independent modernization of RobotReviewer
AU - V. Sahu
AU - A. Sahu
PY - 2026
DA - 2026/09/08
PB - medRxiv
DO - 10.64898/2026.09.08.26362546
UR - https://www.medrxiv.org/content/10.64898/2026.09.08.26362546v1
AB - Evidence-synthesis teams increasingly depend on machine-learning tools to automate risk-of-bias assessment, but these tools frequently rely on unrunnable, deprecated software stacks. Refactoring legacy tools into modern environments is essential for maintenance, yet introduces a critical risk: silently invalidating published performance metrics. Without rigorous validation, systematic reviewers cannot trust that modernized tools retain their predecessors behavior. We present a five-tier validation framework applied to RCT-Reviewer, an independent ground-up modernization of the widely used RobotReviewer system. To prove preservation, we engineered a compatibility shim to execute the original 2017 code alongside the modernized tool using byte-identical weight files. The framework evaluated predictive validity on a human-labelled benchmark, quantified the cost of dropping obsolete ensemble components, proved bit-exact fidelity, tested parser robustness at scale, and included an external validation arm evaluating agreement on 313 trials against a recent reference standard. A control experiment showing 100% agreement between legacy and modernized implementations on identical text proved that variance versus published values is driven by PDF provenance (open-access versus publisher typeset) rather than codebase refactoring. The modernized tool perfectly reproduced original risk-of-bias judgements in 6,018 comparisons. While the screening classifier experienced a bounded performance reduction (F1 0.969 to 0.925) from removing obsolete components, the models core mathematical validity is preserved. These results establish that reviewers can deploy a maintainable modernization with a clearly bounded performance trade-off, offering a reusable validation framework for clinical NLP modernizations. Future work will explore retraining obsolete neural network components to close the performance gap.
ER -