Mechanistic Interpretability of Protein Language Models Reveals Encoded Structural and Functional Properties of Intrinsically Disordered Proteins
- 1University of Cambridge
Abstract
Protein language models (PLMs) such as ESM-2 encode protein sequences as embeddings for downstream tasks. PLMs are trained on a masked learning objective that leverages evolutionary constraints. While interpretability studies of ESM-2 have focused on folded proteins, their behavior on intrinsically disordered proteins (IDPs), which constitute a substantial fraction of the human proteome and are implicated in numerous diseases, remains understudied. Because IDPs experience different types of evolutionary constraints on their amino acid sequences, we hypothesized that PLMs would behave differently on disordered versus folded regions. Here we show that ESM-2 exhibits reduced attention on disordered regions, yet still encodes meaningful biological signals. The model assigns heightened attention to disease-relevant residues even at high levels of disorder. Moreover, we show that both the radius of gyration and individual dynamic contact maps, key characteristics of IDPs, can be obtained from the model logits and embeddings. These findings suggest PLMs capture valuable information relevant to IDP biology despite their bias toward structured residues.
Related
- Structural and Mechanistic Basis of FapB Accessory Functional Amyloid Assembly in Pseudomonas
- Mechanistic Interpretability Needs a Logic of Inference
- Evo 2 as a classification machine: evidence from in-context learning and mechanistic interpretability
- Production of mung bean protein isolates by isoelectric precipitation and ultrafiltration-diafiltration: impact on the composition, structural and techno-functional properties as well as in vitro digestibility
- Impact of S-metolachlor, atrazine and their metabolites on the morphological, structural and functional properties of the intestinal barrier
BibTeX
@misc{naworski2026mechanistic,
title = {{Mechanistic Interpretability of Protein Language Models Reveals Encoded Structural and Functional Properties of Intrinsically Disordered Proteins}},
author = {L. E. Naworski and L. L. Good and R. Scrutton and T. P. J. Knowles},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.10.750691},
url = {https://www.biorxiv.org/content/10.64898/2026.09.10.750691v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Mechanistic Interpretability of Protein Language Models Reveals Encoded Structural and Functional Properties of Intrinsically Disordered Proteins
AU - L. E. Naworski
AU - L. L. Good
AU - R. Scrutton
AU - T. P. J. Knowles
PY - 2026
DA - 2026/09/10
PB - bioRxiv
DO - 10.64898/2026.09.10.750691
UR - https://www.biorxiv.org/content/10.64898/2026.09.10.750691v1
AB - Protein language models (PLMs) such as ESM-2 encode protein sequences as embeddings for downstream tasks. PLMs are trained on a masked learning objective that leverages evolutionary constraints. While interpretability studies of ESM-2 have focused on folded proteins, their behavior on intrinsically disordered proteins (IDPs), which constitute a substantial fraction of the human proteome and are implicated in numerous diseases, remains understudied. Because IDPs experience different types of evolutionary constraints on their amino acid sequences, we hypothesized that PLMs would behave differently on disordered versus folded regions. Here we show that ESM-2 exhibits reduced attention on disordered regions, yet still encodes meaningful biological signals. The model assigns heightened attention to disease-relevant residues even at high levels of disorder. Moreover, we show that both the radius of gyration and individual dynamic contact maps, key characteristics of IDPs, can be obtained from the model logits and embeddings. These findings suggest PLMs capture valuable information relevant to IDP biology despite their bias toward structured residues.
ER -