What survives? Semantic eligibility, distribution shift and metric interpretation in external validation of ADMET predictors
- 1Department of Pharmacy
- 2Daffodil International University
Abstract
Machine-learning models for absorption, distribution, metabolism, excretion and toxicity (ADMET) are usually reported on benchmark splits drawn from the same curated collections used to train them. External validation is rarer, and when it is attempted, the assumption that a public dataset carrying a matching endpoint name is a valid comparator for a model output is seldom examined. We applied a pre-specified, fail-closed semantic eligibility screen — ten declared gates covering biological endpoint, matrix, species, assay family, measurement direction, result semantics, units, transformation, classification threshold and positive class — to twelve candidate pairings between three public experimental datasets and the outputs of ADMET-AI 2.0.1. The screen was executed before any metric was computed. Three pairings were accepted; nine were rejected, each with a recorded reason, and no classification endpoint qualified. On the three accepted pairings, after excluding exact structure overlap with the endpoint training source and with the reconstructed regression multitask universe, performance was poor to moderate: human liver microsomal clearance R² = −0.156 (95% CI −0.217 to −0.108; n = 366), logD at pH 7.4 R² = 0.368 (0.261 to 0.456; n = 474), and human plasma protein binding R² = 0.642 (0.543 to 0.732; n = 178). The model nevertheless outperformed both a training-set-mean baseline and a 1-nearest-neighbour Morgan/Tanimoto baseline on every accepted endpoint, including microsomal clearance, where the mean baseline gave R² = −0.270 and the nearest-neighbour baseline R² = −0.278. Two mechanisms accounted for the pattern. Label-distribution shift between the reconstructed training labels and the external labels was severe for clearance (Kolmogorov–Smirnov D = 0.446) and mild for logD (D = 0.095). Prediction-range compression was extreme for clearance, where predictions had 9% of the observed standard deviation and a calibration slope of 0.037, and moderate for logD (0.781; slope 0.545) and plasma protein binding (0.725; slope 0.599). Chemical novelty explained little (Spearman ρ between maximum training similarity and absolute error, −0.13 to −0.28). Conclusions were unchanged by structure-level aggregation, a structure-clustered bootstrap, restriction to stereochemically unambiguous records, and the inclusion of low-clearance measurements that the original benchmark had excluded before inference. We conclude that assay semantics must be screened before external metrics are computed, and that under strong label-distribution shift R² alone misrepresents model utility; rank correlation, bias, calibration slope and baseline-relative performance should be reported alongside it.
Related
- Machine learning prediction of bismuth recovery from literature data: publication-grouped validation, anchored calibration and chemical interpretation
- 3DRISM Solvation for Accurate and Large-Scale Protein Hydration Prediction and Mechanistic Interpretation
- Chemical Constructions, Mechanism and Biological Interpretation of Nitroreductase-Responsive Systems: From Hypoxia-Associated Imaging to Activatable Therapy
- Reply to Cantwell Chater and Shahsavar: Clarifying PDB deposition records and local model interpretation in GlyT2 structures.
- Integrated Machine Learning and Statistical Interpretation of Malachite Green Removal by SDBS-Modified Ostrich Bone Waste: A Comparative Random Forest, Bayesian Regression, and LMG Analysis
BibTeX
@misc{roktim2026what,
title = {{What survives? Semantic eligibility, distribution shift and metric interpretation in external validation of ADMET predictors}},
author = {Md. Rahul Reza Roktim},
howpublished = {ChemRxiv},
year = {2026},
month = sep,
doi = {10.26434/chemrxiv.15009047/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15009047/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - What survives? Semantic eligibility, distribution shift and metric interpretation in external validation of ADMET predictors
AU - Md. Rahul Reza Roktim
PY - 2026
DA - 2026/09/18
PB - ChemRxiv
DO - 10.26434/chemrxiv.15009047/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15009047/v1
AB - Machine-learning models for absorption, distribution, metabolism, excretion and toxicity (ADMET) are usually reported on benchmark splits drawn from the same curated collections used to train them. External validation is rarer, and when it is attempted, the assumption that a public dataset carrying a matching endpoint name is a valid comparator for a model output is seldom examined. We applied a pre-specified, fail-closed semantic eligibility screen — ten declared gates covering biological endpoint, matrix, species, assay family, measurement direction, result semantics, units, transformation, classification threshold and positive class — to twelve candidate pairings between three public experimental datasets and the outputs of ADMET-AI 2.0.1. The screen was executed before any metric was computed. Three pairings were accepted; nine were rejected, each with a recorded reason, and no classification endpoint qualified. On the three accepted pairings, after excluding exact structure overlap with the endpoint training source and with the reconstructed regression multitask universe, performance was poor to moderate: human liver microsomal clearance R² = −0.156 (95% CI −0.217 to −0.108; n = 366), logD at pH 7.4 R² = 0.368 (0.261 to 0.456; n = 474), and human plasma protein binding R² = 0.642 (0.543 to 0.732; n = 178). The model nevertheless outperformed both a training-set-mean baseline and a 1-nearest-neighbour Morgan/Tanimoto baseline on every accepted endpoint, including microsomal clearance, where the mean baseline gave R² = −0.270 and the nearest-neighbour baseline R² = −0.278. Two mechanisms accounted for the pattern. Label-distribution shift between the reconstructed training labels and the external labels was severe for clearance (Kolmogorov–Smirnov D = 0.446) and mild for logD (D = 0.095). Prediction-range compression was extreme for clearance, where predictions had 9% of the observed standard deviation and a calibration slope of 0.037, and moderate for logD (0.781; slope 0.545) and plasma protein binding (0.725; slope 0.599). Chemical novelty explained little (Spearman ρ between maximum training similarity and absolute error, −0.13 to −0.28). Conclusions were unchanged by structure-level aggregation, a structure-clustered bootstrap, restriction to stereochemically unambiguous records, and the inclusion of low-clearance measurements that the original benchmark had excluded before inference. We conclude that assay semantics must be screened before external metrics are computed, and that under strong label-distribution shift R² alone misrepresents model utility; rank correlation, bias, calibration slope and baseline-relative performance should be reported alongside it.
ER -