Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
Abstract
Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor representations (SRs) make this horizon explicit by encoding discounted distributions over future states. Here, we ask whether such predictive representations can recover not only word classes, but also finer functional and construction-like structure from natural language. A residual network trained on WikiText-103 predicts SR distributions at three horizons without part-of-speech supervision. At the shortest horizon, unsupervised clustering robustly recovers nouns, verbs, and adjectives, while directed inter-cluster transitions reproduce familiar syntactic asymmetries. At finer resolutions and across 13 part-of-speech categories, the same geometry reveals semantic-functional groupings that cross category boundaries and directed relations tracing candidate date, measurement, and title-name constructions. Part-of-speech agreement declines as the predictive horizon lengthens. These results suggest that word classes are coarse regions within a richer predictive geometry in which categorical and construction-like linguistic structure emerge from future-word distributions.
Related
- Retrieval of binding sites across the AlphaFold human proteome using protein language model representations
- Vision-language representations link food perception to subjective valuation
- Comparing serial and parallel language input: Shared neural representations with distinct encoding strengths
- Large language models poorly capture how word order shapes representations of sentence meaning in the human brain.
- Predictive and feedback signals differently shape the formation of language representations
BibTeX
@misc{immertreu2026wordclass,
title = {{Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language}},
author = {Mathis Immertreu and Achim Schilling and Thomas Kinfe and Patrick Krauss},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.24585},
eprint = {2605.24585},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.24585},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
AU - Mathis Immertreu
AU - Achim Schilling
AU - Thomas Kinfe
AU - Patrick Krauss
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2605.24585
UR - https://arxiv.org/abs/2605.24585
AB - Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor representations (SRs) make this horizon explicit by encoding discounted distributions over future states. Here, we ask whether such predictive representations can recover not only word classes, but also finer functional and construction-like structure from natural language. A residual network trained on WikiText-103 predicts SR distributions at three horizons without part-of-speech supervision. At the shortest horizon, unsupervised clustering robustly recovers nouns, verbs, and adjectives, while directed inter-cluster transitions reproduce familiar syntactic asymmetries. At finer resolutions and across 13 part-of-speech categories, the same geometry reveals semantic-functional groupings that cross category boundaries and directed relations tracing candidate date, measurement, and title-name constructions. Part-of-speech agreement declines as the predictive horizon lengthens. These results suggest that word classes are coarse regions within a richer predictive geometry in which categorical and construction-like linguistic structure emerge from future-word distributions.
ER -