A low-rank decoder bottleneck bounds reliability in foundation-model perturbation prediction
Abstract
Foundation-model predictors of perturbation response are evaluated primarily by aggregate accuracy. Using their predictions to prioritize experiments requires assessing each prediction's reliability. We audit the state-transition (ST) decoder used in STATE across three encoders (SE-600M, scGPT, and Tahoe-x1) and four cell lines under CRISPRi. The decoder's first attention layer has a low-rank output. Projecting this output onto 4-32 fixed directions estimated from training activations preserves mean per-perturbation Pearson delta correlation (PDC). This low-rank attention bottleneck bounds the decoder's reliability by distance to training responses: PDC declines as the cosine distance from a perturbation's observed response to its nearest training response increases. We propose predicted-NN-distance, the cosine distance from a predicted response to its nearest training response, as a per-prediction confidence score. The score has a mean absolute Spearman correlation of 0.81 with PDC, close to the 0.88 obtained with the observed response. Thresholds calibrated on this score transfer across encoders and cell lines, and the score remains predictive of PDC for drug perturbations.
Related
- Prediction transfer does not imply reliability transfer in single-cell foundation models
- DNA-LLM2Vec: Architecture-Aware Task-Conditioned Adaptation of Decoder DNA Foundation Models
- Where single-cell foundation models help: a benchmark of annotation, perturbation and regulatory inference against simple baselines
- PerturbBridge: Conditional Latent Schr\"{o}dinger Bridge for Single-Cell Perturbation Prediction
- The recoverable resolution of cellular perturbation-response prediction
BibTeX
@misc{zhang2026lowrank,
title = {{A low-rank decoder bottleneck bounds reliability in foundation-model perturbation prediction}},
author = {Zheyu Zhang and Yuanhao Huang and Fan Feng and Jie Liu},
howpublished = {arXiv},
year = {2026},
month = nov,
doi = {10.48550/arxiv.2611.05849},
eprint = {2611.05849},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.05849},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A low-rank decoder bottleneck bounds reliability in foundation-model perturbation prediction
AU - Zheyu Zhang
AU - Yuanhao Huang
AU - Fan Feng
AU - Jie Liu
PY - 2026
DA - 2026/11/04
PB - arXiv
DO - 10.48550/arxiv.2611.05849
UR - https://arxiv.org/abs/2611.05849
AB - Foundation-model predictors of perturbation response are evaluated primarily by aggregate accuracy. Using their predictions to prioritize experiments requires assessing each prediction's reliability. We audit the state-transition (ST) decoder used in STATE across three encoders (SE-600M, scGPT, and Tahoe-x1) and four cell lines under CRISPRi. The decoder's first attention layer has a low-rank output. Projecting this output onto 4-32 fixed directions estimated from training activations preserves mean per-perturbation Pearson delta correlation (PDC). This low-rank attention bottleneck bounds the decoder's reliability by distance to training responses: PDC declines as the cosine distance from a perturbation's observed response to its nearest training response increases. We propose predicted-NN-distance, the cosine distance from a predicted response to its nearest training response, as a per-prediction confidence score. The score has a mean absolute Spearman correlation of 0.81 with PDC, close to the 0.88 obtained with the observed response. Thresholds calibrated on this score transfer across encoders and cell lines, and the score remains predictive of PDC for drug perturbations.
ER -