Recovery Rates Are Not Comparable Across Transcription Factors: Chance Correction for Attribution Evaluation
Abstract
Attribution methods for genomic sequence models are commonly evaluated by how much of a known motif they recover, or by how a prediction degrades as evidence is deleted. Neither score is interpretable without the value it would take by chance, and neither is routinely reported against one. We show that this omission is not a matter of precision but of validity. The uniform chance level for contiguous motif overlap is L/(N−L+1); across 268 transcription factors in UniBind it ranges from 0.0118 to 0.0427, a 3.6-fold spread determined by motif length and window size alone. For two factors the bootstrap intervals of the chance levels themselves do not overlap, so their raw recovery rates are not comparable quantities. Correcting for this dissolves a published three-way classification of five factors: a factor reported as a resolution failure attains the second-highest corrected value, ahead of one of the two positive controls, and two reported as complete failures fall at or below chance. We further show that perturbation-based evaluation can fail its own precondition: for one factor a fully masked input still scores above the decision boundary, and the curve is not monotone in the number of masked positions, so the area under it is not a measure of faithfulness. We provide chance levels in closed form, a chance-corrected score, and two screens that run before any attribution is computed.
Related
- An Integrated Self-Powered Photoelectrocatalytic System for Quadruple-Function Water Remediation: Organic Degradation, Radionuclide Recycling, Energy Recovery, and Toxicity Evaluation.
- Genome-Wide Identification of Host-Segregating Epidemiological Markers for Source Attribution in Campylobacter jejuni
- Multiple sources, shared readouts: the attribution gap in circular RNA immunotherapy for esophageal squamous cell carcinoma
- An Endothelial Address for the Sphingosine 1-Phosphate Axis: Cell-Type Attribution of 171 Sphingolipid Targets Across 27 Human Cancers
- Highly inducible transcription is impaired in cells lacking Cyclin T1 (CycT1) of the positive transcription factor b (P-TEFb).
BibTeX
@misc{han2026recovery,
title = {{Recovery Rates Are Not Comparable Across Transcription Factors: Chance Correction for Attribution Evaluation}},
author = {Hyunkyung Han and Min Jung Kim},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.16271},
eprint = {2609.16271},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.16271},
}
RIS
TY - UNPB
TI - Recovery Rates Are Not Comparable Across Transcription Factors: Chance Correction for Attribution Evaluation
AU - Hyunkyung Han
AU - Min Jung Kim
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.16271
UR - https://arxiv.org/abs/2609.16271
AB - Attribution methods for genomic sequence models are commonly evaluated by how much of a known motif they recover, or by how a prediction degrades as evidence is deleted. Neither score is interpretable without the value it would take by chance, and neither is routinely reported against one. We show that this omission is not a matter of precision but of validity. The uniform chance level for contiguous motif overlap is L/(N−L+1); across 268 transcription factors in UniBind it ranges from 0.0118 to 0.0427, a 3.6-fold spread determined by motif length and window size alone. For two factors the bootstrap intervals of the chance levels themselves do not overlap, so their raw recovery rates are not comparable quantities. Correcting for this dissolves a published three-way classification of five factors: a factor reported as a resolution failure attains the second-highest corrected value, ahead of one of the two positive controls, and two reported as complete failures fall at or below chance. We further show that perturbation-based evaluation can fail its own precondition: for one factor a fully masked input still scores above the decision boundary, and the curve is not monotone in the number of masked positions, so the area under it is not a measure of faithfulness. We provide chance levels in closed form, a chance-corrected score, and two screens that run before any attribution is computed.
ER -