What a low-bias filter leaves out: measured performance on the excluded negatives of a drug–target interaction benchmark
- 1Center for Biomedical Informatics and Genomics
- 2Tulane University
Abstract
Eligibility rules define which interactions contribute to a benchmark score. The low-bias BindingDB benchmark retains a drug only when it has both positive and negative labels, excluding measured inactive pairs involving drugs without a measured active. We reconstructed this rule in a declared BindingDB population and evaluated retained and excluded negatives against the same positive reference. After cleaning and an additional connectivity-overlap restriction, the excluded region contained 78.4% of the measured negative pairs. DrugBAN and MolTrans were trained in three evaluation designs with five seeds; a released BioT5 checkpoint was evaluated on its published split. Across these evaluations, pooled AUROC was 0.072–0.136 lower on excluded negatives. Lower performance also occurred within proteins with sufficient observations in both regions. DrugBAN remained ahead of MolTrans, but the change in their advantage depended on aggregation: it generally increased under pooled AUROC, whereas equalprotein weighting produced smaller or oppositely directed changes. Cross-protein comparisons accounted for approximately 99.8–99.9% of pooled AUROC's pair-comparison weight. The closer BioT5–DrugBAN comparison additionally depended on scoring convention. These results measure regional performance differences, conditional on the chosen positive reference; they do not estimate a causal filter effect or deployment performance. Benchmark reports should specify eligibility, coverage at the evaluated unit, metric aggregation, and performance on excluded regions where labels are available.
Related
- What actually moves virtual screening performance: two measured resolution limits and a pre-registered decomposition of six interventions
- Performance and Interaction Mechanisms of Bitumen Modified with Devulcanized Rubber-Recycled Polymer Composites
- Measured Host Optical Constants Shift Deep-Blue OLED Outcoupling by up to 14% (Relative) While Leaving the Stack Design Intact
- Understanding the high-resolution spectral signatures of Ar-D2O in the 3OD stretch region measured by cavity ringdown absorption spectroscopy
- Vapor Pressure of Octafluorocyclobutane (RC-318) from 234 K to 387 K Measured with a Variable-Volume View Cell
BibTeX
@misc{li2026what,
title = {{What a low-bias filter leaves out: measured performance on the excluded negatives of a drug–target interaction benchmark}},
author = {Justin Li},
howpublished = {ChemRxiv},
year = {2026},
month = oct,
doi = {10.26434/chemrxiv.15010766/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010766/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - What a low-bias filter leaves out: measured performance on the excluded negatives of a drug–target interaction benchmark
AU - Justin Li
PY - 2026
DA - 2026/10/21
PB - ChemRxiv
DO - 10.26434/chemrxiv.15010766/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010766/v1
AB - Eligibility rules define which interactions contribute to a benchmark score. The low-bias BindingDB benchmark retains a drug only when it has both positive and negative labels, excluding measured inactive pairs involving drugs without a measured active. We reconstructed this rule in a declared BindingDB population and evaluated retained and excluded negatives against the same positive reference. After cleaning and an additional connectivity-overlap restriction, the excluded region contained 78.4% of the measured negative pairs. DrugBAN and MolTrans were trained in three evaluation designs with five seeds; a released BioT5 checkpoint was evaluated on its published split. Across these evaluations, pooled AUROC was 0.072–0.136 lower on excluded negatives. Lower performance also occurred within proteins with sufficient observations in both regions. DrugBAN remained ahead of MolTrans, but the change in their advantage depended on aggregation: it generally increased under pooled AUROC, whereas equalprotein weighting produced smaller or oppositely directed changes. Cross-protein comparisons accounted for approximately 99.8–99.9% of pooled AUROC's pair-comparison weight. The closer BioT5–DrugBAN comparison additionally depended on scoring convention. These results measure regional performance differences, conditional on the chosen positive reference; they do not estimate a causal filter effect or deployment performance. Benchmark reports should specify eligibility, coverage at the evaluated unit, metric aggregation, and performance on excluded regions where labels are available.
ER -