Skip to content
DraconiticResearch
Search Sources API
203,234 studies31 sourcesUpdated 7 min ago
PreprintCC BY 4.0arXiv

In LLM Reasoning, there is Irrationality on top of Value Misalignment

  • Kejiang Qian
  • Fengxiang He
Open at sourcePDFDOICite

Abstract

Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart whose responses maximise utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from bounded prompts, bounded responses, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, Tülu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot avoid, it; (3) self-consistency can improve rationality; and (4) a longer chain of thought improves rationality but with diminishing returns. The code is at https://github.com/EVIEHub/LLM-Rationality

Related

  1. A new proof of the irrationality of e
    Achraf Ben Said
    arXivPreprintarXiv licenceFull textAlgebra & number theoryarXiv
  2. Irrationality of the first stabilization of a complex smooth cubic threefold
    Ting Gong, Shitan Xu
    arXivPreprintarXiv licenceFull textAlgebra & number theoryarXiv
  3. Degree of irrationality of abelian surfaces
    Songsong Huang, Zhi Jiang, Xin Yang
    arXivPreprintCC BY 4.0Full textAlgebra & number theoryarXiv
  4. A Diophantine generalization of Fourier's irrationality trick
    Archan Chattopadhyay, Chandan Chattopadhyay
    arXivPreprintCC BY 4.0Full textAlgebra & number theoryarXiv
  5. Communication and Sensing Coverage of UAV Corridor under Beam Misalignment Error
    Harris K. Armeniakos, Petros S. Bithas, Konstantinos Maliatsos +2
    arXivPreprintCC BY 4.0Full textGeneral mathematicsarXiv

Cite

BibTeXRIS
BibTeX
@misc{qian2026reasoning,
  title = {{In LLM Reasoning, there is Irrationality on top of Value Misalignment}},
  author = {Kejiang Qian and Fengxiang He},
  howpublished = {arXiv},
  year = {2026},
  month = aug,
  doi = {10.48550/arxiv.2606.20624},
  eprint = {2606.20624},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.20624},
  note = {CC BY 4.0},
}
RIS
TY  - UNPB
TI  - In LLM Reasoning, there is Irrationality on top of Value Misalignment
AU  - Kejiang Qian
AU  - Fengxiang He
PY  - 2026
DA  - 2026/08/31
PB  - arXiv
DO  - 10.48550/arxiv.2606.20624
UR  - https://arxiv.org/abs/2606.20624
AB  - Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart whose responses maximise utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from bounded prompts, bounded responses, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, Tülu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot avoid, it; (3) self-consistency can improve rationality; and (4) a longer chain of thought improves rationality but with diminishing returns. The code is at https://github.com/EVIEHub/LLM-Rationality
ER  -