Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations
Abstract
Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.
Related
- Entropy Stable Element-Local Continuous-Discontinuous Residual Correction Summation-By-Parts Discretization
- Quantitative vanishing-regularization limits for degenerate cross-diffusion with changing boundary strata
- Data-Driven MPC with Adaptively Sampled Non-Expert Demonstrations: Performance Guarantees and Sample Complexity
- The Statistical Benefits of Multiple Responses for Learning from Demonstrations
- ICL-SEC: Iterative Cross-Layer Semantic Error Correction
BibTeX
@misc{dhanakshirur2026entropy,
title = {{Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations}},
author = {Mihir Dhanakshirur and Adam Ousherovitch and Ambuj Tewari},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.30572},
eprint = {2609.30572},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.30572},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations
AU - Mihir Dhanakshirur
AU - Adam Ousherovitch
AU - Ambuj Tewari
PY - 2026
DA - 2026/09/24
PB - arXiv
DO - 10.48550/arxiv.2609.30572
UR - https://arxiv.org/abs/2609.30572
AB - Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.
ER -