Identifying and Addressing Systematic Data Leakage in Protein-Ligand Affinity Benchmarks
- 1Enlace Bio
- 2OpenADMET
Abstract
Accurate prediction of protein-ligand binding affinity is a crucial goal in structure-based drug discovery, with the potential to significantly shorten development timelines. Recently, a new wave of machine learning models based on co-folding, such as Boltz-2 and IsoDDE, has demonstrated performance that matches or exceeds that of gold-standard physics-based methods like Free Energy Perturbation (FEP). This paper provides a critical assessment of these claims, revealing that current benchmarks are heavily influenced by data leakage, and proposes a new benchmark that explicitly controls for data leakage. We demonstrate that splitting by protein-sequence identity is inherently insufficient to prevent data leakage due to “target mirroring,” in which homologous proteins with low overall sequence identity still exhibit highly correlated binding profiles. Our meta-analysis of documents in the ChEMBL 36 database identifies more than 6,000 such assay pairs and finds that leakage persists for sequence-identity thresholds as low as 0.2, well below the values commonly used in benchmarks today. Additionally, we show that a ligand-only baseline model, which lacks protein structural information, achieves surprisingly high performance on the FEP+ 4 and OpenFE benchmarks ( r = 0.66 and r = 0.36, respectively). Our results indicate that current benchmarks tend to reward models for memorizing training data and exploiting localized leakage rather than truly learning biophysical principles. To address this issue, we propose the Novelty-Tiered Affinity Benchmark, in which the test data is partitioned into ligand novelty tiers. In the most challenging tier (Tanimoto similarity < 0.35), ligand-only models perform notably worse ( r = 0.14), offering a clear baseline for evaluating genuine generalization. We argue that the field must move beyond sequence-based splits to ensure that AI-driven discovery translates into successful prospective laboratory research.
Related
- Interpreting Public-Dataset SOH Benchmarks: A Systematic Review of Evaluation Protocols and Reproducibility Evidence
- Cassava Starch Thermoplastic Packaging: A Systematic Evidence Map of Additive and Property Relations for Water Affinity and Mechanical Performance
- Reproducibility, Benchmarks, and Mechanistic Diversity: Building Trust in Computational Electrocatalysis for the AI Era
- Accuracy and Efficiency Benchmarks of Pretrained Machine Learning Potentials for Molecular Simulations
- Patient-Level Reassessment of Breast Raman Spectroscopy: Data Leakage and Limits of Subtype Classification
BibTeX
@misc{mattsson2026identifying,
title = {{Identifying and Addressing Systematic Data Leakage in Protein-Ligand Affinity Benchmarks}},
author = {Björn Mattsson and W. Patrick Walters},
howpublished = {bioRxiv (Cold Spring Harbor Laboratory)},
year = {2026},
month = jun,
doi = {10.64898/2026.06.29.735309},
url = {https://doi.org/10.64898/2026.06.29.735309},
note = {CC BY},
}
RIS
TY - UNPB
TI - Identifying and Addressing Systematic Data Leakage in Protein-Ligand Affinity Benchmarks
AU - Björn Mattsson
AU - W. Patrick Walters
PY - 2026
DA - 2026/06/30
PB - bioRxiv (Cold Spring Harbor Laboratory)
DO - 10.64898/2026.06.29.735309
UR - https://doi.org/10.64898/2026.06.29.735309
AB - Accurate prediction of protein-ligand binding affinity is a crucial goal in structure-based drug discovery, with the potential to significantly shorten development timelines. Recently, a new wave of machine learning models based on co-folding, such as Boltz-2 and IsoDDE, has demonstrated performance that matches or exceeds that of gold-standard physics-based methods like Free Energy Perturbation (FEP). This paper provides a critical assessment of these claims, revealing that current benchmarks are heavily influenced by data leakage, and proposes a new benchmark that explicitly controls for data leakage. We demonstrate that splitting by protein-sequence identity is inherently insufficient to prevent data leakage due to “target mirroring,” in which homologous proteins with low overall sequence identity still exhibit highly correlated binding profiles. Our meta-analysis of documents in the ChEMBL 36 database identifies more than 6,000 such assay pairs and finds that leakage persists for sequence-identity thresholds as low as 0.2, well below the values commonly used in benchmarks today. Additionally, we show that a ligand-only baseline model, which lacks protein structural information, achieves surprisingly high performance on the FEP+ 4 and OpenFE benchmarks ( r = 0.66 and r = 0.36, respectively). Our results indicate that current benchmarks tend to reward models for memorizing training data and exploiting localized leakage rather than truly learning biophysical principles. To address this issue, we propose the Novelty-Tiered Affinity Benchmark, in which the test data is partitioned into ligand novelty tiers. In the most challenging tier (Tanimoto similarity < 0.35), ligand-only models perform notably worse ( r = 0.14), offering a clear baseline for evaluating genuine generalization. We argue that the field must move beyond sequence-based splits to ensure that AI-driven discovery translates into successful prospective laboratory research.
ER -