Generalization properties of the token perceptron: the role of embedding and vocabulary
Abstract
Large language models are trained to predict tokens from a finite vocabulary, each represented by a vector in a d-dimensional embedding space. Therefore, token prediction is a classification problem whose classes have a geometry determined by the embedding. We study how this geometry affects learning in a minimal solvable model: a linear predictor that maps a sequence of N embedded tokens to a d-dimensional output through one d× d matrix per position, trained in the teacher--student setting. We call this the \textit{token perceptron}. We compute its generalization error with the replica method, taking N→∞ at fixed number of examples per position α=P/N and keeping d and the vocabulary size D finite. As a first reference case, we consider continuous embeddings on the sphere, which isolate the role of the embedding dimension. The natural load is then α/d, and for d>1 a normalized label fixes the direction of the teacher output, so the problem becomes a noiseless regression: the teacher is recovered exactly above αrec=d2/(d−1), and spherical cross-entropy reaches this threshold at large inverse temperature. In the same setting the storage capacity of random associations scales as d2 for margins O(1/√d). We then turn to vocabulary embeddings, where a label only identifies the Voronoi cell of the teacher output, so that each example imposes inequalities rather than equations, as in the classical perceptron. We compute the Bayes-optimal error for a given embedding matrix through an approximate spectral closure and show that the optimal learner first resolves the directions in which the vocabulary has the largest variance, even when the teacher is isotropic.
Related
- Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization
- Free energy fluctuations of symmetric perceptron models up to and including criticality
- Strong freezing in binary perceptron models
- A Robust Perceptron Cycling Theorem and Applications
- Vocabulary Growth Fundamentals: Bernstein Functions and Hausdorff Sequences
BibTeX
@misc{bocchi2026generalization,
title = {{Generalization properties of the token perceptron: the role of embedding and vocabulary}},
author = {Dario Bocchi and Matteo Negri and Flavio Nicoletti and Francesco D'Amico},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.32788},
eprint = {2610.32788},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.32788},
note = {CC BY-SA 4.0},
}
RIS
TY - UNPB
TI - Generalization properties of the token perceptron: the role of embedding and vocabulary
AU - Dario Bocchi
AU - Matteo Negri
AU - Flavio Nicoletti
AU - Francesco D'Amico
PY - 2026
DA - 2026/10/23
PB - arXiv
DO - 10.48550/arxiv.2610.32788
UR - https://arxiv.org/abs/2610.32788
AB - Large language models are trained to predict tokens from a finite vocabulary, each represented by a vector in a d-dimensional embedding space. Therefore, token prediction is a classification problem whose classes have a geometry determined by the embedding. We study how this geometry affects learning in a minimal solvable model: a linear predictor that maps a sequence of N embedded tokens to a d-dimensional output through one d× d matrix per position, trained in the teacher--student setting. We call this the \textit{token perceptron}. We compute its generalization error with the replica method, taking N→∞ at fixed number of examples per position α=P/N and keeping d and the vocabulary size D finite. As a first reference case, we consider continuous embeddings on the sphere, which isolate the role of the embedding dimension. The natural load is then α/d, and for d>1 a normalized label fixes the direction of the teacher output, so the problem becomes a noiseless regression: the teacher is recovered exactly above α_rec=d^2/(d−1), and spherical cross-entropy reaches this threshold at large inverse temperature. In the same setting the storage capacity of random associations scales as d^2 for margins O(1/√d). We then turn to vocabulary embeddings, where a label only identifies the Voronoi cell of the teacher output, so that each example imposes inequalities rather than equations, as in the classical perceptron. We compute the Bayes-optimal error for a given embedding matrix through an approximate spectral closure and show that the optimal learner first resolves the directions in which the vocabulary has the largest variance, even when the teacher is isotropic.
ER -