Cerberus: bidirectional state space blocks improve accuracy and efficiency of regulatory sequence models
- 1Calico Life Sciences LLC
Abstract
Sequence-to-function models predict regulatory activity and variant effects from DNA sequence alone, and leading models such as Borzoi compute long-range interactions with self-attention. The reference genome caps unique training sequences, and new assays add labels, not sequences. This constraint favors architectures that learn more from limited sequence diversity. We compared long-range blocks with the rest of the architecture and the training data held fixed, replacing Borzoi's transformer blocks with Hydra, a bidirectional state space block built on Mamba-2. Hydra blocks ran 5.7-fold faster on 8,192-token inputs, and models built on them predicted binned coverage on held-out sequences and classified fine-mapped eQTLs more accurately. Hydra's linear cost in sequence length let us run the blocks at 32 bp resolution and remove the U-net decoder, leaving a convolution-SSM model smaller and more accurate than the one it replaced. Scaling this architecture up on an augmented track collection produced Cerberus, an ensemble of eight models over 786 kb of input. We evaluated it on GTEx v11 fine-mapped eQTL, sQTL, and paQTL benchmarks with negatives matched on allele frequency, gene expression, and phenotype-specific positional annotations to reduce confounding by these properties. Cerberus is more accurate than Borzoi on all three phenotypes, 0.692 versus 0.668 mean eQTL AUPRC across 48 tissues, and estimates eQTL effect sizes better, 0.379 versus 0.321 Spearman ρ. Against AlphaGenome, it trades leads in eQTL classification, ahead on variants 3 to 100 kb from the transcription start site and behind in coding sequence and within 3 kb. Interpretation of the trained blocks shows interaction range growing with block depth, from a 0.9 kb median half-life in the first block to 79 kb in the seventh, and the deepest heads anchoring on promoters, enhancers, and CTCF peaks. We release the models, the training data, the training and evaluation code, and the benchmark sets, supporting variant scoring and transfer learning.
Related
- Bi-Mamba-Load: A bidirectional structured state space model with sequence relationship adaptive perception for electricity load forecasting
- The glucocorticoid receptor–autophagy axis in inflammation, immunoregulation, and cancer: a bidirectional regulatory network with therapeutic potential
- Sequence-aware genotyping improves the accuracy of molecular markers associated with cassava mosaic disease resistance loci
- Computational Prediction of Recombinant Expression Efficiency for Industrial Enzymes: An Integrated Co-Translational Sequence Analysis Platform
- Evidence for a regulatory role of the evolutionarily conserved sequence complementarity between tRNAs
BibTeX
@misc{kelley2026cerberus,
title = {{Cerberus: bidirectional state space blocks improve accuracy and efficiency of regulatory sequence models}},
author = {D. R. Kelley and H. Yuan and X. Huang and J. Linder},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.02.756377},
url = {https://www.biorxiv.org/content/10.64898/2026.10.02.756377v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Cerberus: bidirectional state space blocks improve accuracy and efficiency of regulatory sequence models
AU - D. R. Kelley
AU - H. Yuan
AU - X. Huang
AU - J. Linder
PY - 2026
DA - 2026/10/02
PB - bioRxiv
DO - 10.64898/2026.10.02.756377
UR - https://www.biorxiv.org/content/10.64898/2026.10.02.756377v1
AB - Sequence-to-function models predict regulatory activity and variant effects from DNA sequence alone, and leading models such as Borzoi compute long-range interactions with self-attention. The reference genome caps unique training sequences, and new assays add labels, not sequences. This constraint favors architectures that learn more from limited sequence diversity. We compared long-range blocks with the rest of the architecture and the training data held fixed, replacing Borzoi's transformer blocks with Hydra, a bidirectional state space block built on Mamba-2. Hydra blocks ran 5.7-fold faster on 8,192-token inputs, and models built on them predicted binned coverage on held-out sequences and classified fine-mapped eQTLs more accurately. Hydra's linear cost in sequence length let us run the blocks at 32 bp resolution and remove the U-net decoder, leaving a convolution-SSM model smaller and more accurate than the one it replaced. Scaling this architecture up on an augmented track collection produced Cerberus, an ensemble of eight models over 786 kb of input. We evaluated it on GTEx v11 fine-mapped eQTL, sQTL, and paQTL benchmarks with negatives matched on allele frequency, gene expression, and phenotype-specific positional annotations to reduce confounding by these properties. Cerberus is more accurate than Borzoi on all three phenotypes, 0.692 versus 0.668 mean eQTL AUPRC across 48 tissues, and estimates eQTL effect sizes better, 0.379 versus 0.321 Spearman ρ. Against AlphaGenome, it trades leads in eQTL classification, ahead on variants 3 to 100 kb from the transcription start site and behind in coding sequence and within 3 kb. Interpretation of the trained blocks shows interaction range growing with block depth, from a 0.9 kb median half-life in the first block to 79 kb in the seventh, and the deepest heads anchoring on promoters, enhancers, and CTCF peaks. We release the models, the training data, the training and evaluation code, and the benchmark sets, supporting variant scoring and transfer learning.
ER -