Complex Problem Solving in Large Language Models: A Statistical Control Survey and Diagnostic Framework
Abstract
Complex problem solving (CPS) with large language models (LLMs) is often framed as a matter of stronger reasoning or longer generation. Yet early-step error amplification, prompt brittleness, and failures to revise incorrect commitments are difficult to explain by missing knowledge or expressive capacity alone. This survey interprets CPS as a sequential estimation-and-decision problem over a latent solution state. A controller maintains a belief about an unobserved solution trajectory, updates it as noisy intermediate evidence arrives, and decides whether to commit, verify, branch, roll back, or abstain to minimize expected loss. Reasoning supplies candidate transitions and interpretations, whereas process control shapes and evaluates those proposals and regulates subsequent transitions and observations. Within this framework, we organize existing methods around five components: explicit state representation, transition structuring, validation and constraint enforcement, search and rollback, and uncertainty management. We also interpret evaluation metrics according to the statistical quantities they estimate. The framework further yields a diagnostic hypothesis: interventions should be most effective when they target the error or uncertainty component implicated by an observed failure. We distinguish systematic, stochastic, and irreducible error together with epistemic and aleatoric uncertainty, and call this alignment problem-control fit and its failure control mismatch. For example, additional sampling may reduce sampling variability while leaving a shared systematic error unchanged. This perspective clarifies what current methods estimate and control, what remains uncontrolled, and why reliable validation, targeted recovery, calibrated uncertainty, and matched-budget evaluation are central open problems.
Related
- MIMO and Multi-Group Comparison Problem in Process Control: A Multivariate Statistical Framework for Systems Represented with Frequency Response Functions
- On the relaxation problem in statistical mechanics
- A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
- Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation
- When and Why Do Linear Bias Probes Fail? A Geometric and Statistical Theory of Bias Detectability in Large Language Model Representations
BibTeX
@misc{cai2026complex,
title = {{Complex Problem Solving in Large Language Models: A Statistical Control Survey and Diagnostic Framework}},
author = {Jiazhang Cai and Tao Wang and Ruidong Zhang and Siyuan Li and Terry Ma and Luyang Fang and Haoran Lu and Huimin Cheng and Yingchuan Zhang and Shushan Wu and Rui Xie and Lin Tang and Chao Huang and Rongjie Liu and Ziyu Liu and Meizhi Yu and Yongkai Chen and Yifan Zhou and Zeliang Sun and Chang Liu and Zhen Xiang and Wei Xiao and Zixin Rao and Xinyi Liu and Yutong Hu and Mengrui Zhang and Jing Zhang and Weidi Luo and Jincheng Yu and Zhengliang Liu and Weihang You and Hanqi Jiang and Yi Pan and Junhao Chen and Xinliang Li and Tianming Liu and Wenxuan Zhong and Ping Ma},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.20973},
eprint = {2609.20973},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.20973},
}
RIS
TY - UNPB
TI - Complex Problem Solving in Large Language Models: A Statistical Control Survey and Diagnostic Framework
AU - Jiazhang Cai
AU - Tao Wang
AU - Ruidong Zhang
AU - Siyuan Li
AU - Terry Ma
AU - Luyang Fang
AU - Haoran Lu
AU - Huimin Cheng
AU - Yingchuan Zhang
AU - Shushan Wu
AU - Rui Xie
AU - Lin Tang
AU - Chao Huang
AU - Rongjie Liu
AU - Ziyu Liu
AU - Meizhi Yu
AU - Yongkai Chen
AU - Yifan Zhou
AU - Zeliang Sun
AU - Chang Liu
AU - Zhen Xiang
AU - Wei Xiao
AU - Zixin Rao
AU - Xinyi Liu
AU - Yutong Hu
AU - Mengrui Zhang
AU - Jing Zhang
AU - Weidi Luo
AU - Jincheng Yu
AU - Zhengliang Liu
AU - Weihang You
AU - Hanqi Jiang
AU - Yi Pan
AU - Junhao Chen
AU - Xinliang Li
AU - Tianming Liu
AU - Wenxuan Zhong
AU - Ping Ma
PY - 2026
DA - 2026/09/17
PB - arXiv
DO - 10.48550/arxiv.2609.20973
UR - https://arxiv.org/abs/2609.20973
AB - Complex problem solving (CPS) with large language models (LLMs) is often framed as a matter of stronger reasoning or longer generation. Yet early-step error amplification, prompt brittleness, and failures to revise incorrect commitments are difficult to explain by missing knowledge or expressive capacity alone. This survey interprets CPS as a sequential estimation-and-decision problem over a latent solution state. A controller maintains a belief about an unobserved solution trajectory, updates it as noisy intermediate evidence arrives, and decides whether to commit, verify, branch, roll back, or abstain to minimize expected loss. Reasoning supplies candidate transitions and interpretations, whereas process control shapes and evaluates those proposals and regulates subsequent transitions and observations. Within this framework, we organize existing methods around five components: explicit state representation, transition structuring, validation and constraint enforcement, search and rollback, and uncertainty management. We also interpret evaluation metrics according to the statistical quantities they estimate. The framework further yields a diagnostic hypothesis: interventions should be most effective when they target the error or uncertainty component implicated by an observed failure. We distinguish systematic, stochastic, and irreducible error together with epistemic and aleatoric uncertainty, and call this alignment problem-control fit and its failure control mismatch. For example, additional sampling may reduce sampling variability while leaving a shared systematic error unchanged. This perspective clarifies what current methods estimate and control, what remains uncontrolled, and why reliable validation, targeted recovery, calibrated uncertainty, and matched-budget evaluation are central open problems.
ER -