TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

Abstract

Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing -- a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, on the Meta-World+ MT50 benchmark, TOPPO achieves stronger mean and tail-task performance than prior SAC-family and ARS-family methods while using substantially fewer parameters and reaching their final performance in fewer environment steps. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOPPO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.

Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
Saunak Kumar Panda, Tong Li, Yisha Xiang +1
arXiv13 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv
Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
Kunqi Huang, Lican Kang, Hongrui Shu +1
arXiv2 Nov 2026PreprintarXiv licenceFull textStatisticsarXiv
Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data
Joonhyeok Jeong, Jy-yong Sohn, Kibok Lee
arXiv23 Oct 2026PreprintarXiv licenceFull textStatisticsarXiv
Learning Adaptive SED for heterogeneous load balancing
Sanne van Kempen, Jaron Sanders, Fiona Sloothaak +1
arXiv7 Sep 2026PreprintCC BY 4.0Full textProbabilityarXiv
Distributional Balancing with Machine Learning for Clinical Trial Augmentation Using Real-World Data
Zern Ke, Mingshi Cui, Gemma Moran +1
arXiv20 Sep 2026PreprintarXiv licenceFull textStatisticsarXiv

Cite

BibTeX RIS

BibTeX

@misc{li2026toppo,
  title = {{TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing}},
  author = {Yuanpeng Li and Gefei Lin and Annie Qu and Rui Miao},
  howpublished = {arXiv},
  year = {2026},
  month = nov,
  doi = {10.48550/arxiv.2605.11473},
  eprint = {2605.11473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.11473},
  note = {CC BY-NC-SA 4.0},
}

RIS

TY  - UNPB
TI  - TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
AU  - Yuanpeng Li
AU  - Gefei Lin
AU  - Annie Qu
AU  - Rui Miao
PY  - 2026
DA  - 2026/11/03
PB  - arXiv
DO  - 10.48550/arxiv.2605.11473
UR  - https://arxiv.org/abs/2605.11473
AB  - Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing -- a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, on the Meta-World+ MT50 benchmark, TOPPO achieves stronger mean and tail-task performance than prior SAC-family and ARS-family methods while using substantially fewer parameters and reaching their final performance in fewer environment steps. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOPPO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.
ER  -

Abstract

Related

Cite