Optimality of delayed reward attainment under uncertainty
Abstract
Decision making under uncertainty requires balancing the value of additional information against the cost of delaying the decision until that information becomes available. This challenge is especially prevalent for embodied decisions such as those faced by animals moving between food sources and robots selecting navigation targets, where movement through space shapes both the quality of information and the cost of acting. We consider a version of this problem, where the decision is on choosing among rewards of uncertain value, and those rewards can only be attained through changing the decision maker's state sufficiently. Our focus is on interactions between reward learning and state evolution, restated as a deterministic optimal control problem with two terminal alternatives, one with a known reward and the other learned gradually through noisy observations. We demonstrate that delayed reward attainment is often optimal in a variety of systems, especially so when the alternatives are initially difficult to distinguish and the cost of waiting is relatively low. For a subset of problems, we also derive sufficient conditions that allow shortening the planning horizon a priori, guaranteeing that further observations would not affect the value function. Finally, when observation quality depends on the state, we show that it is often optimal for a controller to evolve the state strategically toward more informative sensing locations before making a decision, in line with the behavior broadly observed in biological and engineered systems.
Related
- Achieving an O(1/N) Optimality Gap in Average-Reward Weakly-Coupled MDPs
- Norm attainment for multilinear operators and polynomials on Banach Spaces and Banach lattices
- A bi-Lipschitz characterization of strong minimum-attainment for Lipschitz maps
- Attainment of the sharp stability constant of Beckner conformal log-Sobolev inequality
- Finite attainment and efficient computation of regularized quantum channel relative entropy
BibTeX
@misc{wang2026optimality,
title = {{Optimality of delayed reward attainment under uncertainty}},
author = {Chuwei Wang and Alexander Vladimirsky and Anastasia Bizyaeva},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.03506},
eprint = {2610.03506},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.03506},
}
RIS
TY - UNPB
TI - Optimality of delayed reward attainment under uncertainty
AU - Chuwei Wang
AU - Alexander Vladimirsky
AU - Anastasia Bizyaeva
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.03506
UR - https://arxiv.org/abs/2610.03506
AB - Decision making under uncertainty requires balancing the value of additional information against the cost of delaying the decision until that information becomes available. This challenge is especially prevalent for embodied decisions such as those faced by animals moving between food sources and robots selecting navigation targets, where movement through space shapes both the quality of information and the cost of acting. We consider a version of this problem, where the decision is on choosing among rewards of uncertain value, and those rewards can only be attained through changing the decision maker's state sufficiently. Our focus is on interactions between reward learning and state evolution, restated as a deterministic optimal control problem with two terminal alternatives, one with a known reward and the other learned gradually through noisy observations. We demonstrate that delayed reward attainment is often optimal in a variety of systems, especially so when the alternatives are initially difficult to distinguish and the cost of waiting is relatively low. For a subset of problems, we also derive sufficient conditions that allow shortening the planning horizon a priori, guaranteeing that further observations would not affect the value function. Finally, when observation quality depends on the state, we show that it is often optimal for a controller to evolve the state strategically toward more informative sensing locations before making a decision, in line with the behavior broadly observed in biological and engineered systems.
ER -