Adaptive Agent Design
Abstract
We consider an agent acting against a general non-Markovian environment. The agent maintains its agent states, but is free to choose a transition kernel across those states and optimize its state-feedback control policies. We study the bi-level agent design problem that optimizes the transition kernel and the policy it induces, given said kernel with offline data of observations and actions obtained via a behavioral policy. For general environments, we show that a soft Q-learning algorithm converges almost surely to the fixed point of a soft Bellman equation defined by the stationary averages that the behavioral policy and the chosen kernel induce, and we delineate what separates the resulting policy from an optimal one. In partially observed Markov decision problems, we analyze convergence properties of parametrized transition kernel design via zero-th order and Bayesian optimization techniques.
Related
- PEAR: Permutation-Equivariant Adaptive Routing Multi-Agent Debate
- A Path-Dependent Agent-Based Microsimulation of Crime and Violence Reduction Policy Portfolios in Bolivia: Survey Calibration, Adaptive Emulator Ensembles, Global Sensitivity, Tempered MCMC, and Multi-Objective Policy Analysis
- Principal-agent hypothesis testing under adaptive experimentation
- SiteEvo: An LLM Agent for Autonomous Discovery of Site-Adaptive Massive MIMO Detectors
- FACTS-MAS: A Multi-Agent Architecture for Adaptive Forecasting from Heterogeneous Evidence
BibTeX
@misc{velicheti2026adaptive,
title = {{Adaptive Agent Design}},
author = {Raj Kiriti Velicheti and Subhonmesh Bose and Tamer Başar},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.12486},
eprint = {2609.12486},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.12486},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Adaptive Agent Design
AU - Raj Kiriti Velicheti
AU - Subhonmesh Bose
AU - Tamer Başar
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.12486
UR - https://arxiv.org/abs/2609.12486
AB - We consider an agent acting against a general non-Markovian environment. The agent maintains its agent states, but is free to choose a transition kernel across those states and optimize its state-feedback control policies. We study the bi-level agent design problem that optimizes the transition kernel and the policy it induces, given said kernel with offline data of observations and actions obtained via a behavioral policy. For general environments, we show that a soft Q-learning algorithm converges almost surely to the fixed point of a soft Bellman equation defined by the stationary averages that the behavioral policy and the chosen kernel induce, and we delineate what separates the resulting policy from an optimal one. In partially observed Markov decision problems, we analyze convergence properties of parametrized transition kernel design via zero-th order and Bayesian optimization techniques.
ER -