Learn Your Own Thoughts: Abstract Token Curriculum
Abstract
Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT techniques require explicit supervision on thinking tokens, which requires rich, task-specific data. In this work, we propose Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design. ATC gradually increases problem complexity through a sequence of distributions, training the model to develop internal abstract ``thoughts'' in the continuous representation space. This paper provides both theoretical and experimental evidence for the benefits of ATC and its advantages over previous methods for training continuous thoughts. Theoretically, we show that for learning parity functions with single-layer softmax attention using ATC, attention naturally focuses on the CoT tokens in the context that provide the ``easiest path'' to predicting the next token. Experimentally, we show ATC's effectiveness on graph reachability and arithmetic learning tasks.
Related
- Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models
- New thoughts on fractional Sobolev spaces
- Thoughts on Quantum Events, Quantizing Gravity and von Neumann Algebras
- When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning
- On abstract commensurators of groups
BibTeX
@misc{gatmiry2026learn,
title = {{Learn Your Own Thoughts: Abstract Token Curriculum}},
author = {Khashayar Gatmiry and Avrajit Ghosh and Parsa Mirtaheri and Jason D. Lee and Nika Haghtalab and Emmanuel Abbe and Peter Bartlett},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.19717},
eprint = {2609.19717},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.19717},
}
RIS
TY - UNPB
TI - Learn Your Own Thoughts: Abstract Token Curriculum
AU - Khashayar Gatmiry
AU - Avrajit Ghosh
AU - Parsa Mirtaheri
AU - Jason D. Lee
AU - Nika Haghtalab
AU - Emmanuel Abbe
AU - Peter Bartlett
PY - 2026
DA - 2026/09/23
PB - arXiv
DO - 10.48550/arxiv.2609.19717
UR - https://arxiv.org/abs/2609.19717
AB - Large Language Models (LLMs) have achieved remarkable reasoning capabilities by utilizing chain-of-thought (CoT) as a scratchpad for intermediate stages of thinking. However, CoT techniques require explicit supervision on thinking tokens, which requires rich, task-specific data. In this work, we propose Abstract Token Curriculum (ATC), a novel curriculum learning framework that elicits effective continuous intermediate representations without direct supervision or manual scratchpad design. ATC gradually increases problem complexity through a sequence of distributions, training the model to develop internal abstract ``thoughts'' in the continuous representation space. This paper provides both theoretical and experimental evidence for the benefits of ATC and its advantages over previous methods for training continuous thoughts. Theoretically, we show that for learning parity functions with single-layer softmax attention using ATC, attention naturally focuses on the CoT tokens in the context that provide the ``easiest path'' to predicting the next token. Experimentally, we show ATC's effectiveness on graph reachability and arithmetic learning tasks.
ER -