What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content
Abstract
Entropy over chain-of-thought tokens decides which tokens receive the policy gradient, which get pruned, and whether a run has collapsed, yet each such statistic reads a next-token distribution mixing three choices: whether to emit connective scaffolding, which connective, and what the substantive continuation should be. Designating a scaffold vocabulary subset separates the three, exactly, for entropy, Kullback--Leibler divergence, and the first-order entropy velocity of a softmax policy. We prove the raw and content conventions disagree about which position is the larger fork on an explicit open region, and bound answer diversity by the content channel plus a leakage term a measured witness certifies. Across twenty-three configurations the scaffold side carries up to 41% of the raw high-entropy set; on a matched-tokenizer ladder, coupling changes only at the math-corpus step while the scaffold's entropy share keeps growing through distillation; a closed-form forecast from one channel correlation tracks selection retention over a 54-point range to five points, unfitted. On compression, the content convention beats raw surprisal in every cell; an answer-leakage audit then corrects our own headline control: re-fed chains earn a quarter to a half of their accuracy from restated answers, and once stripped, no token scorer beats a random contiguous block.
Related
- Measure of Maximal Entropy for Geodesic Flows on Noncompact Manifolds of Nonpositive Curvature
- Finite attainment and efficient computation of regularized quantum channel relative entropy
- Renyi's Relative Entropy Lyapunov Functional for Convergence to Invariant Measure in Dissipative Stochastic Hamiltonian Systems
- Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel
- Fixed-Point Scaffolding in the Clef Programming Language
BibTeX
@misc{papamichalis2026what,
title = {{What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content}},
author = {Marios Papamichalis and Regina Ruane},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.25039},
eprint = {2609.25039},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.25039},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - What Does Chain-of-Thought Entropy Measure? A Channel Audit of Scaffolding, Routing, and Content
AU - Marios Papamichalis
AU - Regina Ruane
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.25039
UR - https://arxiv.org/abs/2609.25039
AB - Entropy over chain-of-thought tokens decides which tokens receive the policy gradient, which get pruned, and whether a run has collapsed, yet each such statistic reads a next-token distribution mixing three choices: whether to emit connective scaffolding, which connective, and what the substantive continuation should be. Designating a scaffold vocabulary subset separates the three, exactly, for entropy, Kullback--Leibler divergence, and the first-order entropy velocity of a softmax policy. We prove the raw and content conventions disagree about which position is the larger fork on an explicit open region, and bound answer diversity by the content channel plus a leakage term a measured witness certifies. Across twenty-three configurations the scaffold side carries up to 41% of the raw high-entropy set; on a matched-tokenizer ladder, coupling changes only at the math-corpus step while the scaffold's entropy share keeps growing through distillation; a closed-form forecast from one channel correlation tracks selection retention over a 54-point range to five points, unfitted. On compression, the content convention beats raw surprisal in every cell; an answer-leakage audit then corrects our own headline control: re-fed chains earn a quarter to a half of their accuracy from restated answers, and once stripped, no token scorer beats a random contiguous block.
ER -