Rigorous nested cross-validation and external generalization-gap analysis of a machine-learning classifier for U-251 glioblastoma cytotoxicity
- 1School of Pharmacy, Hyogo Medical University, Kobe, Hyogo, Japan
- 2Department of Medical Molecular Informatics, Meiji Pharmaceutical University, Kiyose, Tokyo, Japan
Abstract
Glioblastoma remains difficult to treat, and drug-repositioning strategies require efficient methods to evaluate chemical libraries. Here, we developed a machine-learning binary classifier for compound activity against U-251 glioblastoma cells using a corrected single-center dataset of 236 compounds evaluated under a standardized cell-viability assay. Compounds were labeled as active or inactive using a predefined viability criterion (<50% vs. ≥ 50%) within a single-point screening framework using 10 µM as the upper screening concentration. To reduce optimistic bias and information leakage, we used repeated nested cross-validation in which preprocessing, feature selection, and hyperparameter optimization were performed within the training folds. ExtraTrees was selected based on Matthews correlation coefficient, showing moderate internal performance (ROC-AUC, 0.76; balanced accuracy, 0.66; MCC, 0.36). Structural analyses using nearest-neighbor Tanimoto similarity and Bemis–Murcko scaffolds indicated that the training set was scaffold-diverse with limited series bias. However, when the locked model was applied to an independent external set of 53 compounds, all compounds were classified as inactive, resulting in zero sensitivity, balanced accuracy of 0.5, and MCC of 0. The external ROC-AUC based on continuous prediction scores was 0.37, indicating poor threshold-independent ranking performance. Nearest-neighbor Tanimoto similarity did not explain prediction correctness or rescue active-compound recovery. Y-randomization supported the presence of a non-random internal signal, whereas scaffold-grouped cross-validation suggested that internal ROC-AUC was not solely driven by scaffold leakage. Overall, this study provides a reusable dataset and reproducible workflow, and highlights a substantial generalization gap between internal validation and external prediction in small chemical machine-learning datasets.
Related
- A richness-sensitive Acoustic Biodiversity Index from species-level detections: classifier benchmarking, false-positive control and validation against standard soundscape indices in an African savanna and an urban zoo
- Rigorous validation of graph-based network analysis reappraises biologically coherent ADHD-associated transcriptomic modules in peripheral blood
- Homology-aware cross-validation strategies for generalization assessment in RNA secondary structure prediction
- A formation-response classifier for borehole deviation using conventional well log data
- Enhancing Clinical Classifier Robustness with Deep Representation Learning Based OOD Detection on DNA Methylation Data
BibTeX
@article{uemichi2026rigorous,
title = {{Rigorous nested cross-validation and external generalization-gap analysis of a machine-learning classifier for U-251 glioblastoma cytotoxicity}},
author = {Yuki Uemichi and Shuhei Nakao and Shuji Nagano and Miyuki Mabuchi and Yuya Kawabata and Yoshihiro Uesawa and Tadashi Shimizu},
journal = {PLOS One},
year = {2026},
month = nov,
doi = {10.1371/journal.pone.0361912},
url = {https://doi.org/10.1371/journal.pone.0361912},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Rigorous nested cross-validation and external generalization-gap analysis of a machine-learning classifier for U-251 glioblastoma cytotoxicity
AU - Yuki Uemichi
AU - Shuhei Nakao
AU - Shuji Nagano
AU - Miyuki Mabuchi
AU - Yuya Kawabata
AU - Yoshihiro Uesawa
AU - Tadashi Shimizu
PY - 2026
DA - 2026/11/03
JO - PLOS One
DO - 10.1371/journal.pone.0361912
UR - https://doi.org/10.1371/journal.pone.0361912
AB - Glioblastoma remains difficult to treat, and drug-repositioning strategies require efficient methods to evaluate chemical libraries. Here, we developed a machine-learning binary classifier for compound activity against U-251 glioblastoma cells using a corrected single-center dataset of 236 compounds evaluated under a standardized cell-viability assay. Compounds were labeled as active or inactive using a predefined viability criterion (<50% vs. ≥ 50%) within a single-point screening framework using 10 µM as the upper screening concentration. To reduce optimistic bias and information leakage, we used repeated nested cross-validation in which preprocessing, feature selection, and hyperparameter optimization were performed within the training folds. ExtraTrees was selected based on Matthews correlation coefficient, showing moderate internal performance (ROC-AUC, 0.76; balanced accuracy, 0.66; MCC, 0.36). Structural analyses using nearest-neighbor Tanimoto similarity and Bemis–Murcko scaffolds indicated that the training set was scaffold-diverse with limited series bias. However, when the locked model was applied to an independent external set of 53 compounds, all compounds were classified as inactive, resulting in zero sensitivity, balanced accuracy of 0.5, and MCC of 0. The external ROC-AUC based on continuous prediction scores was 0.37, indicating poor threshold-independent ranking performance. Nearest-neighbor Tanimoto similarity did not explain prediction correctness or rescue active-compound recovery. Y-randomization supported the presence of a non-random internal signal, whereas scaffold-grouped cross-validation suggested that internal ROC-AUC was not solely driven by scaffold leakage. Overall, this study provides a reusable dataset and reproducible workflow, and highlights a substantial generalization gap between internal validation and external prediction in small chemical machine-learning datasets.
SN - 1932-6203
ER -