Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets
Abstract
This paper studies differential properties of objective functions for discrete-time discounted Markov decision processes (MDPs) with finite state and action sets, when differentiation is considered with respect to the discount factor. Given a fixed discount factor for an MDP, we estimate the norms and seminorms of the objective functions, their derivatives, and the value function at a different discount factor. We then use these estimates to prove that objective functions, their derivatives, and value functions are Lipschitz continuous in discount factors. As a result, if an MDP is solved with a fixed discount factor, then we can estimate the value function at a different discount factor without solving it again.\\ %Following these estimations, we also reveal the relations between the seminorms of objective functions, their behavior when the discount factor approaches $1,$ and the discount factors at which the set of optimal policies changes.
Related
- Linear Occupancy Homomorphisms for Discounted Markov Decision Processes
- A sequential quadratic programming method for optimization with expectation objective functions, deterministic inequality constraints and robust subproblems
- Simple Extensions of Single-Objective Acquisition Functions and Hedge Strategies for Multi-Objective Bayesian Optimization
- Synthesis of the Transient Behaviour of Optimization Algorithms for Strongly Convex Objective Functions
- High-order fractional linear differential equations do not describe stochastic processes with finite state space
BibTeX
@misc{feinberg2026differential,
title = {{Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets}},
author = {Eugene A. Feinberg and Gaojin He},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.39326},
eprint = {2610.39326},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.39326},
}
RIS
TY - UNPB
TI - Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets
AU - Eugene A. Feinberg
AU - Gaojin He
PY - 2026
DA - 2026/10/29
PB - arXiv
DO - 10.48550/arxiv.2610.39326
UR - https://arxiv.org/abs/2610.39326
AB - This paper studies differential properties of objective functions for discrete-time discounted Markov decision processes (MDPs) with finite state and action sets, when differentiation is considered with respect to the discount factor. Given a fixed discount factor for an MDP, we estimate the norms and seminorms of the objective functions, their derivatives, and the value function at a different discount factor. We then use these estimates to prove that objective functions, their derivatives, and value functions are Lipschitz continuous in discount factors. As a result, if an MDP is solved with a fixed discount factor, then we can estimate the value function at a different discount factor without solving it again.\\ %Following these estimations, we also reveal the relations between the seminorms of objective functions, their behavior when the discount factor approaches $1,$ and the discount factors at which the set of optimal policies changes.
ER -