LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
Abstract
Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.
Related
- Efficient structural information integration into protein language models with implicit mixture-of-experts ProBEiT
- A structure-informed deep learning framework for modeling TCR-peptide-HLA interactions
- SaltyMeta: a curated benchmark and protein language model-informed web tool for salty peptide prediction
- Evolution-Informed Viroinformatics Identifies Conserved Immune Targets with Predicted Lower Mutational Vulnerability Across Betacoronaviruses.
- Evolution towards higher unitary conductance in mammals makes BK channels more efficient and precise
BibTeX
@misc{banerjee2026lemonzest,
title = {{LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling}},
author = {Biswajit Banerjee and Claudia Alvarez Carreno and Anton S. Petrov},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.37675},
eprint = {2609.37675},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37675},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
AU - Biswajit Banerjee
AU - Claudia Alvarez Carreno
AU - Anton S. Petrov
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.37675
UR - https://arxiv.org/abs/2609.37675
AB - Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.
ER -