MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
5 affiliations
- 1Massachusetts Institute of Technology
- 2Czech Institute of Informatics , Robotics and Cybernetics , Czech Technical University in Prague
- 3Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences
- 4Enveda Biosciences
- 5Tufts University
Abstract
Reliable benchmarking is critical for developing machine learning models for tandem mass spectrometry (MS/MS) based molecule discovery. Subtle issues in experimental design and model evaluation procedures can degrade the trustworthiness of such benchmarks and lead to erroneous conclusions. We conduct a thorough review of model evaluation issues in the recent MS/MS machine learning literature, using the standard MassSpecGym benchmark suite as a case study to illustrate the impact of these issues. We find evaluation issues in at least 17 of 26 papers reporting MassSpecGym benchmark results in the first year of its adoption. We isolate three classes of failures: (i) data leakage, (ii) shortcut learning, and (iii) implementation bugs and metric divergence. Through extensive experimentation and code replication, we quantify the impact of these issues and show how they corrupt the evaluation standards MassSpecGym was designed to enforce. We distill our findings into recommendations generalizable to MS/MS challenges, benchmarks, and custom evaluation setups. We also release MassSpecGym v1.5, an implementation of our recommendations in the MassSpecGym benchmarking suite which addresses the failure modes identified in this audit. MassSpecGym v1.5 is publicly available at https://github.com/pluskal-lab/MassSpecGym.
Related
- A Scoping Review of Diffusion-Based De Novo Molecular Generation: Architectures, Evaluation, and Open Challenges for Computational Drug Discovery
- Reliability of AI Methods in Drug Discovery: Evaluation of Boltz-2 for Structure and Binding Affinity Prediction
- Discovery of novel vanillin-2-acetylfuran hybrids through integrated design, synthesis, computational studies, and in-vitro cytotoxic evaluation
- Generative discovery, symmetry-aware membership screening, and device-level evaluation of six-member-ring MoS2 polytypes
- Discovery of Two Potential and Novel Toll-like Receptor 7 Antagonists via Molecular Docking, Molecular Dynamics Simulation and In Vitro Biological Evaluation
BibTeX
@misc{liu2026massspecgym,
title = {{MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery}},
author = {Hongxuan Liu and Roman Bushuiev and Ivy Lightheart and Mrunali Manjrekar and Anton Bushuiev and Magdalena Lederbauer and Filip Jozefov and Yinkai Wang and Soha Hassoun and Josef Šivic and James Taylor and Runzhong Wang and David Healey and Tomáš Pluskal and Connor W. Coley},
howpublished = {arXiv (Cornell University)},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.19624},
url = {https://arxiv.org/abs/2606.19624},
note = {CC BY},
}
RIS
TY - UNPB
TI - MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
AU - Hongxuan Liu
AU - Roman Bushuiev
AU - Ivy Lightheart
AU - Mrunali Manjrekar
AU - Anton Bushuiev
AU - Magdalena Lederbauer
AU - Filip Jozefov
AU - Yinkai Wang
AU - Soha Hassoun
AU - Josef Šivic
AU - James Taylor
AU - Runzhong Wang
AU - David Healey
AU - Tomáš Pluskal
AU - Connor W. Coley
PY - 2026
DA - 2026/06/17
PB - arXiv (Cornell University)
DO - 10.48550/arxiv.2606.19624
UR - https://arxiv.org/abs/2606.19624
AB - Reliable benchmarking is critical for developing machine learning models for tandem mass spectrometry (MS/MS) based molecule discovery. Subtle issues in experimental design and model evaluation procedures can degrade the trustworthiness of such benchmarks and lead to erroneous conclusions. We conduct a thorough review of model evaluation issues in the recent MS/MS machine learning literature, using the standard MassSpecGym benchmark suite as a case study to illustrate the impact of these issues. We find evaluation issues in at least 17 of 26 papers reporting MassSpecGym benchmark results in the first year of its adoption. We isolate three classes of failures: (i) data leakage, (ii) shortcut learning, and (iii) implementation bugs and metric divergence. Through extensive experimentation and code replication, we quantify the impact of these issues and show how they corrupt the evaluation standards MassSpecGym was designed to enforce. We distill our findings into recommendations generalizable to MS/MS challenges, benchmarks, and custom evaluation setups. We also release MassSpecGym v1.5, an implementation of our recommendations in the MassSpecGym benchmarking suite which addresses the failure modes identified in this audit. MassSpecGym v1.5 is publicly available at https://github.com/pluskal-lab/MassSpecGym.
SN - 2331-8422
ER -