What topology hold-out removes—and what it does not
- 1College of Chemical Engineering
- 2Nanjing Tech University
Abstract
Machine-learning screens of metal–organic frameworks (MOFs) for CO2 capture increasingly report the validation protocol that produced a score, and topology hold-out has become the standard stricter alternative to random splitting. Prior work already reads topology hold-out as a combined structural and chemical distribution shift; this work measures the size of the chemistry that the split withholds. On the hypothetical-MOF library hMOF (324,426 structures), 74.7% of coarse-grained chemical compositions occur in a single topology, accounting for 61.3% of structures, and seven of eleven topologies leave every structure without a chemical counterpart in training. On the one usable chemistry-matched pair (pcu ↔nbo), removing the partner topology raised nbo-side MAE by 70.0% while a size-matched control did not, and removing only the shared-composition partner rows improved pcu-side MAE, so partner coverage is direction-dependent rather than uniformly beneficial. A chemistry-key hold-out produces only 21.1% of the topology hold-out's R2 decrease, so the support loss is real but does not translate proportionally into performance loss. Across five independently redrawn 25,000-sample support sets, the top-1% hit rate fell from 50.4% to 19.3% under topology-grouped validation. A support audit in ARC-MOF, which is partly dependent on hMOF and uses different composition keys, confirms strong coupling under bounded sensitivity rather than as an independent replication. We retain topology hold-out as a stress test while recommending that groupedsplit reports disclose their chemical support, deployment-relevant aggregation and ranking performance.
Related
- A grouped split silently removes the chemistry entangled with its grouping variable: the case of MOF topology hold-out
- What the Overpotential Does Not Tell You: Diagnosing Non-Comparability Across Hydrogen and Oxygen Evolution Benchmarking Metrics
- What Does a Semilocal Machine-Learning Correction Actually Learn? Size-Dependent Errors across Four Parent Functionals
- What Cycle Closure Can and Cannot See in a Relative Binding Free Energy Network
- PFAS Toxicity: What’s True, What’s Not, and What Really Matters
BibTeX
@misc{wang2026what,
title = {{What topology hold-out removes—and what it does not}},
author = {Tao Wang},
howpublished = {ChemRxiv},
year = {2026},
month = oct,
doi = {10.26434/chemrxiv.15010923/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010923/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - What topology hold-out removes—and what it does not
AU - Tao Wang
PY - 2026
DA - 2026/10/23
PB - ChemRxiv
DO - 10.26434/chemrxiv.15010923/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010923/v1
AB - Machine-learning screens of metal–organic frameworks (MOFs) for CO2 capture increasingly report the validation protocol that produced a score, and topology hold-out has become the standard stricter alternative to random splitting. Prior work already reads topology hold-out as a combined structural and chemical distribution shift; this work measures the size of the chemistry that the split withholds. On the hypothetical-MOF library hMOF (324,426 structures), 74.7% of coarse-grained chemical compositions occur in a single topology, accounting for 61.3% of structures, and seven of eleven topologies leave every structure without a chemical counterpart in training. On the one usable chemistry-matched pair (pcu ↔nbo), removing the partner topology raised nbo-side MAE by 70.0% while a size-matched control did not, and removing only the shared-composition partner rows improved pcu-side MAE, so partner coverage is direction-dependent rather than uniformly beneficial. A chemistry-key hold-out produces only 21.1% of the topology hold-out's R2 decrease, so the support loss is real but does not translate proportionally into performance loss. Across five independently redrawn 25,000-sample support sets, the top-1% hit rate fell from 50.4% to 19.3% under topology-grouped validation. A support audit in ARC-MOF, which is partly dependent on hMOF and uses different composition keys, confirms strong coupling under bounded sensitivity rather than as an independent replication. We retain topology hold-out as a stress test while recommending that groupedsplit reports disclose their chemical support, deployment-relevant aggregation and ranking performance.
ER -