Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample

@misc{li2026does,
  title = {{Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample}},
  author = {Wenye Li and Ke Wei},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.19853},
  eprint = {2610.19853},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19853},
  note = {CC BY 4.0},
}

TY  - UNPB
TI  - Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample
AU  - Wenye Li
AU  - Ke Wei
PY  - 2026
DA  - 2026/10/14
PB  - arXiv
DO  - 10.48550/arxiv.2610.19853
UR  - https://arxiv.org/abs/2610.19853
AB  - Policy mirror descent (PMD) refers to a class of policy optimization methods associated with different mirror maps. For a broad class of decomposable Legendre mirror maps encompassing many standard choices, Li and Wei [2026] show that the PMD policy sequence converges to an optimal policy from any admissible initialization under any positive constant step size, even when the optimal policy is not unique. In this note, we provide a counterexample with assistance from OpenAI's GPT-5.6 model, showing that PMD with a non-decomposable Legendre mirror map need not converge in policy.
ER  -