A theory of platonic representations in language models
Abstract
Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
Related
- When and Why Do Linear Bias Probes Fail? A Geometric and Statistical Theory of Bias Detectability in Large Language Model Representations
- Laplace--King representations: density and spectral theory
- Character theory for 2-representations
- Algebraic Theory of Inductive Mathematical Modeling: From Experimental Observations to State-Space Representations
- Lorentz Hyperbolic Weighted Regression: Theory for Fixed and Estimated Representations
BibTeX
@misc{doshi2026theory,
title = {{A theory of platonic representations in language models}},
author = {Darshil Doshi and Wenjie Zhou and Corinna Elena Wegner and Daniel J. Korchinski and Santiago Acevedo and Matthieu Wyart},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07168},
eprint = {2610.07168},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07168},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A theory of platonic representations in language models
AU - Darshil Doshi
AU - Wenjie Zhou
AU - Corinna Elena Wegner
AU - Daniel J. Korchinski
AU - Santiago Acevedo
AU - Matthieu Wyart
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.07168
UR - https://arxiv.org/abs/2610.07168
AB - Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.
ER -