{
  "abstract": "Protein language models are trained primarily with masked language modeling (MLM), which predicts masked amino-acid identities. Joint-embedding predictive architectures (JEPA) instead predict latent representations, but have not been applied to proteins. ProteinJEPA supplements MLM with a cosine loss for predicting the half-depth hidden states of a teacher given the unmasked sequence. On 19 tasks, with ESM2 at 35M and 150M parameters and three pretraining seeds, MLM+JEPA outperforms compute-matched and step-matched MLM-only continued training in 78 and 76 of 114 comparisons (14 losses, 22 ties). The median compute-matched gain is $+0.0106$ on structure- and homology-sensitive tasks versus $+0.0041$ elsewhere, led by SCOPe-40 retrieval and remote homology with improvements of 6.1 percentage points in Recall@1 and 2.7 points in accuracy, respectively. Gains on these tasks increase with model size from 8M to 150M. Against the off-the-shelf checkpoint, MLM+JEPA wins 81 of 114 comparisons (median $+0.0068$) without improving MLM loss. In random initialization the gain is smaller and replicates inconsistently across seeds ($p{=}0.059$). The same recipe improves the causal ProGen3 model, beating a compute-matched next-token-prediction control on 12 of 16 tasks. Ablations show that cosine loss beats mean squared error, while adding shallower targets removes most of the task gain. JEPA-only training collapses downstream performance: latent prediction complements MLM rather than replacing it. Code: https://anonymous.4open.science/r/protJepa-FF24",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2605.07554",
  "author_total": 3,
  "authors": [
    {
      "name": "Dan Ofer"
    },
    {
      "name": "Dafna Shahaf"
    },
    {
      "name": "Michal Linial"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/frzqhah82h3we.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2605.07554",
  "fulltext_url": null,
  "id": "frzqhah82h3we",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2605.07554",
  "language": null,
  "licence": "CC-BY-NC-SA-4.0",
  "licence_label": "CC BY-NC-SA 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "q-bio.BM"
    ],
    [
      "arxiv",
      "stat.ML"
    ],
    [
      "openalex",
      "1312|Machine Learning in Bioinformatics"
    ]
  ],
  "openalex": "W7160837003",
  "pdf_url": "https://arxiv.org/pdf/2605.07554",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-23",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/frzqhah82h3we.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:24:01Z",
      "native_id": "2605.07554",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2605.07554"
    }
  ],
  "subjects": [
    {
      "field": "biochemistry",
      "origin": "arxiv:q-bio.BM",
      "subfield": "biophysics"
    },
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    }
  ],
  "title": "ProteinJEPA: Latent prediction improves protein language model pretraining",
  "title_text": "ProteinJEPA: Latent prediction improves protein language model pretraining",
  "type": "preprint",
  "updated_at": "2026-10-30T12:24:06Z",
  "url": "https://research.draconitic.com/paper/frzqhah82h3we",
  "venue": "arXiv",
  "versions": []
}