Active Inference as a Convex Markov Decision Process
Abstract
Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). This perspective reveals that policy-dependent reward prediction errors transmit natural gradients of the expected free energy backwards in time rather than up a hierarchy. Finally, we show that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning. Together this places EFE minimization within modern reinforcement learning and optimization theory and opens a route toward principled algorithms for active inference.
Related
- Convex-Analytic View of Query Selection for Safety Verification of Markov Decision Processes
- Robust Markov Decision Processes: An Introduction Using Convex Optimization and Game Theory
- Worst-Case Bayesian CVaR Regret Bounds for Episodic Markov Decision Processes
- Dynamic Programming for Value-at-Risk Markov Decision Processes in Continuous Spaces
- Percolation Markov Decision Processes
BibTeX
@misc{milosevic2026active,
title = {{Active Inference as a Convex Markov Decision Process}},
author = {Nikola Milosevic and Nicolás Hinrichs and Nico Scherf},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2607.20152},
eprint = {2607.20152},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.20152},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Active Inference as a Convex Markov Decision Process
AU - Nikola Milosevic
AU - Nicolás Hinrichs
AU - Nico Scherf
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2607.20152
UR - https://arxiv.org/abs/2607.20152
AB - Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). This perspective reveals that policy-dependent reward prediction errors transmit natural gradients of the expected free energy backwards in time rather than up a hierarchy. Finally, we show that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning. Together this places EFE minimization within modern reinforcement learning and optimization theory and opens a route toward principled algorithms for active inference.
ER -