Metagenomic contextualization of proteins with state space models
- 1Oregon State University
Abstract
Genomic foundation models have emerged as powerful tools for learning representations of biological sequences, but existing approaches predominantly rely on Transformer architectures, whose quadratic attention complexity limits their ability to capture long-range genomic context. This limitation is particularly relevant for metagenomics, where sequencing data have increased more than 500-fold since 2011, generating increasingly large and diverse datasets. For example, gLM2, a Transformer-based mixed-modality genomic language model, was trained with a context window of 4,096 tokens, limiting its ability to capture extended genomic context within metagenomic contigs. Developing scalable architectures capable of processing longer sequences while integrating protein and nucleotide information is therefore essential for advancing genomic foundation models. We introduce Nammu, a 167-million-parameter, bidirectional Mamba-based genomic foundation model that integrates protein and nucleotide sequences within a 20,000-token context window. Leveraging the linear scaling of state-space models, Nammu was trained using a curriculum learning strategy on the OpenMetaGenomic (OMG) corpus, first incorporating 64 million protein sequences followed by 32 million mixed-modality metagenomic contigs. Despite using approximately 37\% fewer training tokens than gLM2, Nammu outperformed this Transformer-based model in taxonomic classification across all evaluated taxonomic ranks on a marine metagenomic dataset from the Critical Assessment of Metagenome Interpretation (CAMI). On the exact-feature-unmatched CAMI II Strain Madness test set, Nammu achieved higher all-contig genus-classification accuracy than Kraken2 (0.9182 versus 0.8142). Nammu further outperformed gLM2 (150M) in KEGG Orthology prediction using deep-sea metagenome-assembled genomes. Together, these results demonstrate the potential of state-space architectures for scalable, multimodal genomic representation learning and highlight their ability to support accurate taxonomic and functional predictions from metagenomic sequences.
Related
- City life: airborne DNA metagenomic biodiversity monitoring reveals dynamic changes across time and space
- MEG-Mamba: A Scalable State-Space Foundation Model for Magnetoencephalography
- Bi-Mamba-Load: A bidirectional structured state space model with sequence relationship adaptive perception for electricity load forecasting
- ViSSR: Physics-Informed Hybrid Vision-LSTM/State-Space Super-Resolution for Digital Rock Microstructure Analysis
- Efficient Convex Optimization Methods for State-Space Heartbeat Dynamics Models with Gamma Generalized Linear Models
BibTeX
@misc{azbijari2026metagenomic,
title = {{Metagenomic contextualization of proteins with state space models}},
author = {N. Azbijari and J. H. Wynne and K. Irizarry and T. Sistla and A. R. Thurber and M. David},
howpublished = {bioRxiv},
year = {2026},
month = jul,
doi = {10.64898/2026.07.07.736993},
url = {https://www.biorxiv.org/content/10.64898/2026.07.07.736993v2},
note = {CC BY},
}
RIS
TY - UNPB
TI - Metagenomic contextualization of proteins with state space models
AU - N. Azbijari
AU - J. H. Wynne
AU - K. Irizarry
AU - T. Sistla
AU - A. R. Thurber
AU - M. David
PY - 2026
DA - 2026/07/07
PB - bioRxiv
DO - 10.64898/2026.07.07.736993
UR - https://www.biorxiv.org/content/10.64898/2026.07.07.736993v2
AB - Genomic foundation models have emerged as powerful tools for learning representations of biological sequences, but existing approaches predominantly rely on Transformer architectures, whose quadratic attention complexity limits their ability to capture long-range genomic context. This limitation is particularly relevant for metagenomics, where sequencing data have increased more than 500-fold since 2011, generating increasingly large and diverse datasets. For example, gLM2, a Transformer-based mixed-modality genomic language model, was trained with a context window of 4,096 tokens, limiting its ability to capture extended genomic context within metagenomic contigs. Developing scalable architectures capable of processing longer sequences while integrating protein and nucleotide information is therefore essential for advancing genomic foundation models. We introduce Nammu, a 167-million-parameter, bidirectional Mamba-based genomic foundation model that integrates protein and nucleotide sequences within a 20,000-token context window. Leveraging the linear scaling of state-space models, Nammu was trained using a curriculum learning strategy on the OpenMetaGenomic (OMG) corpus, first incorporating 64 million protein sequences followed by 32 million mixed-modality metagenomic contigs. Despite using approximately 37\% fewer training tokens than gLM2, Nammu outperformed this Transformer-based model in taxonomic classification across all evaluated taxonomic ranks on a marine metagenomic dataset from the Critical Assessment of Metagenome Interpretation (CAMI). On the exact-feature-unmatched CAMI II Strain Madness test set, Nammu achieved higher all-contig genus-classification accuracy than Kraken2 (0.9182 versus 0.8142). Nammu further outperformed gLM2 (150M) in KEGG Orthology prediction using deep-sea metagenome-assembled genomes. Together, these results demonstrate the potential of state-space architectures for scalable, multimodal genomic representation learning and highlight their ability to support accurate taxonomic and functional predictions from metagenomic sequences.
ER -