Optimality of Threshold Policies for Heat Storage Optimization and a Policy Gradient Algorithm
Abstract
We study the optimal control of a stylized finite-horizon heat-storage system operated by a prosumer who can buy heat from and sell heat to a district heating network. Heat production and consumption are uncertain, and charging and discharging decisions have to be taken before the subsequent heat balance is observed. The resulting problem is formulated as a Markov decision process with continuous state and action spaces. We first establish a structural characterization of the optimal policy as a two-parameter threshold policy. Under additional regularity assumptions, we prove differentiability and Lipschitz properties of the value functions. Exploiting the threshold structure, we parameterize the policy by the time-dependent lower and upper thresholds and derive a pathwise stochastic-gradient estimator. We then propose a projected stochastic policy-gradient algorithm, establish a constrained Kurdyka-Łojasiewicz inequality and derive global finite-sample bounds for both the objective optimality gap and the distance to the optimal threshold vector. Finally, numerical experiments for a non-stationary heat-balance process illustrate the evolution of the threshold parameters.
Related
- An Adaptive Projected-Gradient Algorithm for Sample-Average Approximations of Stochastic Multi-Objective Optimization
- A Particle-Swarm-Assisted Gradient Meta-Learning Algorithm for Joint Transmit Precoding and STAR-RIS Coefficient Optimization
- Optimality of Affine Policies in Distributionally Robust Linear-Quadratic Control with Temporally Correlated Noise
- Optimal long-run control of endemic infections: bang-bang threshold policies in a stochastic SIS model
- Minimax optimality for sequential gradient-free minimization of smooth functions and their derivatives
BibTeX
@misc{bauerle2026optimality,
title = {{Optimality of Threshold Policies for Heat Storage Optimization and a Policy Gradient Algorithm}},
author = {Nicole Bäuerle and Florian Döttling},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.37853},
eprint = {2610.37853},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.37853},
}
RIS
TY - UNPB
TI - Optimality of Threshold Policies for Heat Storage Optimization and a Policy Gradient Algorithm
AU - Nicole Bäuerle
AU - Florian Döttling
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.37853
UR - https://arxiv.org/abs/2610.37853
AB - We study the optimal control of a stylized finite-horizon heat-storage system operated by a prosumer who can buy heat from and sell heat to a district heating network. Heat production and consumption are uncertain, and charging and discharging decisions have to be taken before the subsequent heat balance is observed. The resulting problem is formulated as a Markov decision process with continuous state and action spaces. We first establish a structural characterization of the optimal policy as a two-parameter threshold policy. Under additional regularity assumptions, we prove differentiability and Lipschitz properties of the value functions. Exploiting the threshold structure, we parameterize the policy by the time-dependent lower and upper thresholds and derive a pathwise stochastic-gradient estimator. We then propose a projected stochastic policy-gradient algorithm, establish a constrained Kurdyka-Łojasiewicz inequality and derive global finite-sample bounds for both the objective optimality gap and the distance to the optimal threshold vector. Finally, numerical experiments for a non-stationary heat-balance process illustrate the evolution of the threshold parameters.
ER -