Self-Architecting Protein Transformers: An Empirical Study
- 1University of Modena and Reggio Emilia
Abstract
Motivation. Protein language models (pLMs) such as ESM-2 and ProtBERT rely on pretraining corpora of tens to hundreds of millions of sequences and on encoder architectures whose depth, width and number of attention heads are chosen by the practitioner and never revisited during training. The entry cost of state-of-the-art pLMs is therefore out of reach for laboratories without industrial-scale infrastructure, and the fixed architecture provides no in-training diagnostic of whether the chosen capacity matches the structural complexity of the data. This work asks whether a self-architecting transformer, which grows its own width and depth from quantitative signals derived from the attention matrices, can extract competitive protein representations from a single reference proteome. Results. A three-level self-architecting framework, INCRT-geo, is applied to masked-language pretraining on the human Ensembl proteome (approximately twenty thousand sequences). On Pfam-50 family classification, the principal model attains a linear-probe accuracy that exceeds two pretrained baselines, ESM-2 small and ProtBERT, despite a corpus several orders of magnitude smaller. Three single-variable ablations isolate the contributions of one-residue tokenisation, depth growth and an asymmetry-loss regulariser; the regulariser is shown to be necessary for the depth-growth trigger to fire. Scaling pretraining to eight vertebrate proteomes does not improve Pfam accuracy under the available compute budget; the negative result is reported transparently. Architectural diagnostics indicate that the heads allocated by the framework are functionally diverse rather than redundant.
Related
- PRINT: A semi-empirical canopy water removal approach for improving estimates of leaf nitrogen and grain protein contents in winter wheat
- A protein–foldamer supramolecular synthon for self-assembled hybrid architectures
- Rationally Engineered Self-Assembling Protein Nanoparticles for Allergen-Specific Peanut Allergy Immunotherapy
- Deep-learning-based design of an orthogonal self-labeling protein from K-Ras(G12C)
- BiHC: Protein domain segmentation by bidirectional hierarchical clustering using Voronoi tessellation and physical self-containment
BibTeX
@misc{cirrincione2026selfarchitecting,
title = {{Self-Architecting Protein Transformers: An Empirical Study}},
author = {G. Cirrincione and E. Ficarra and M. Lovino},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.09.750410},
url = {https://www.biorxiv.org/content/10.64898/2026.09.09.750410v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Self-Architecting Protein Transformers: An Empirical Study
AU - G. Cirrincione
AU - E. Ficarra
AU - M. Lovino
PY - 2026
DA - 2026/09/09
PB - bioRxiv
DO - 10.64898/2026.09.09.750410
UR - https://www.biorxiv.org/content/10.64898/2026.09.09.750410v1
AB - Motivation. Protein language models (pLMs) such as ESM-2 and ProtBERT rely on pretraining corpora of tens to hundreds of millions of sequences and on encoder architectures whose depth, width and number of attention heads are chosen by the practitioner and never revisited during training. The entry cost of state-of-the-art pLMs is therefore out of reach for laboratories without industrial-scale infrastructure, and the fixed architecture provides no in-training diagnostic of whether the chosen capacity matches the structural complexity of the data. This work asks whether a self-architecting transformer, which grows its own width and depth from quantitative signals derived from the attention matrices, can extract competitive protein representations from a single reference proteome. Results. A three-level self-architecting framework, INCRT-geo, is applied to masked-language pretraining on the human Ensembl proteome (approximately twenty thousand sequences). On Pfam-50 family classification, the principal model attains a linear-probe accuracy that exceeds two pretrained baselines, ESM-2 small and ProtBERT, despite a corpus several orders of magnitude smaller. Three single-variable ablations isolate the contributions of one-residue tokenisation, depth growth and an asymmetry-loss regulariser; the regulariser is shown to be necessary for the depth-growth trigger to fire. Scaling pretraining to eight vertebrate proteomes does not improve Pfam accuracy under the available compute budget; the negative result is reported transparently. Architectural diagnostics indicate that the heads allocated by the framework are functionally diverse rather than redundant.
ER -