SMILES serialization changes activity-cliff annotations in the MoleculeACE benchmark
- 1Faculty of Engineering
- 2King Mongkut’s University of Technology Thonburi (KMUTT
Abstract
MoleculeACE is a widely used benchmark that annotates activity-cliff compounds in 30 ChEMBL datasets and reports that machine-learning models predict them poorly. Its “soft consensus” criterion admits a compound pair when any of three similarities reaches 0.9: ECFP4 Tanimoto similarity, Tanimoto similarity of anonymized molecular graphs, or normalized Levenshtein similarity between SMILES strings. The first two are computed after parsing; the third operates on the supplied strings, which the benchmark code does not canonicalize. This study held molecules, potencies, thresholds and code fixed and varied only how the same molecules werewritten. A re-implementation reproduced the distributed labels exactly in 27 of 30 datasets, and canonical round-tripping changed amedian of 0.00% of labels. Re-serializing themolecules with two independent cheminformatics toolkits, with every rewritten string verified to encode the original molecule, changed a median of 8.98% (Indigo) and 9.48% (Open Babel) of compound labels; Kekulé SMILES from RDKit changed 5.23%, and randomized SMILES 17.05%. The graph-derived criteria were invariant across all rewrites. Because MoleculeACE stratifies its train/test split on the cliff label, changing the labels caused split membership to approach that of an independent redraw (31.3% of compounds reassigned, against a 32.21% ceiling). An end-to-end repair, with serialization-invariant labels, re-derived splits and retrained descriptorbased models, moved the pooled cliff penalty by −0.006 to +0.020 log units; every confidence interval included zero against a minimum detectable effect of 0.041–0.057, and for SVM the shift fell within the variability of same-definition split redraws of comparable magnitude. The distributed annotation is therefore reproducible but not invariant to valid serialization. Benchmarks that retain string-based criteria should specify and enforce the canonicalization contract (toolkit, version and output mode); otherwise, labels should be defined on parsed molecular representations. Serialization-invariant labels, splits and per-molecule predictions are released.
Related
- From Optical Transitions Through Morphology Changes to Charge Transport: A P3HT Benchmark Study Using the Critical Point Model
- Rescuable Stagnation: A Diagnostic for Benchmark Suitability in LLM-Guided Materials Optimization
- Accuracy, Reliability, and Efficiency of a MACE Machine-Learning Interatomic Potential for Aspirin: A Reproducible Benchmark for Molecular Simulation
- An Audited Open Record of Three Deep-Blue TADF Host Materials with Benchmark Tasks for Machine Learning
- Surfactant-Toolkit and SurfBench-200: an open computation toolkit and a benchmark of tool-augmented language-model reasoning in surfactant science
BibTeX
@misc{sriwicha2026smiles,
title = {{SMILES serialization changes activity-cliff annotations in the MoleculeACE benchmark}},
author = {Phongsakon Sriwicha},
howpublished = {ChemRxiv},
year = {2026},
month = oct,
doi = {10.26434/chemrxiv.15009988/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15009988/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - SMILES serialization changes activity-cliff annotations in the MoleculeACE benchmark
AU - Phongsakon Sriwicha
PY - 2026
DA - 2026/10/06
PB - ChemRxiv
DO - 10.26434/chemrxiv.15009988/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15009988/v1
AB - MoleculeACE is a widely used benchmark that annotates activity-cliff compounds in 30 ChEMBL datasets and reports that machine-learning models predict them poorly. Its “soft consensus” criterion admits a compound pair when any of three similarities reaches 0.9: ECFP4 Tanimoto similarity, Tanimoto similarity of anonymized molecular graphs, or normalized Levenshtein similarity between SMILES strings. The first two are computed after parsing; the third operates on the supplied strings, which the benchmark code does not canonicalize. This study held molecules, potencies, thresholds and code fixed and varied only how the same molecules werewritten. A re-implementation reproduced the distributed labels exactly in 27 of 30 datasets, and canonical round-tripping changed amedian of 0.00% of labels. Re-serializing themolecules with two independent cheminformatics toolkits, with every rewritten string verified to encode the original molecule, changed a median of 8.98% (Indigo) and 9.48% (Open Babel) of compound labels; Kekulé SMILES from RDKit changed 5.23%, and randomized SMILES 17.05%. The graph-derived criteria were invariant across all rewrites. Because MoleculeACE stratifies its train/test split on the cliff label, changing the labels caused split membership to approach that of an independent redraw (31.3% of compounds reassigned, against a 32.21% ceiling). An end-to-end repair, with serialization-invariant labels, re-derived splits and retrained descriptorbased models, moved the pooled cliff penalty by −0.006 to +0.020 log units; every confidence interval included zero against a minimum detectable effect of 0.041–0.057, and for SVM the shift fell within the variability of same-definition split redraws of comparable magnitude. The distributed annotation is therefore reproducible but not invariant to valid serialization. Benchmarks that retain string-based criteria should specify and enforce the canonicalization contract (toolkit, version and output mode); otherwise, labels should be defined on parsed molecular representations. Serialization-invariant labels, splits and per-molecule predictions are released.
ER -