Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations
Abstract
Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.
Related
- Multi-Agent Reinforcement Learning for Movable Antenna-aided Cell-Free Massive MIMO Systems
- Computation and Applications of Euclidean and Normed Representations of Massive Data
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
- Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
BibTeX
@misc{makkar2026towards,
title = {{Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations}},
author = {Aditya Makkar and Benjamin Unger and Jeongyeol Kwon and Mathieu Laurière and Eugene Vinitsky and Yonathan Efroni},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.02928},
eprint = {2609.02928},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.02928},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations
AU - Aditya Makkar
AU - Benjamin Unger
AU - Jeongyeol Kwon
AU - Mathieu Laurière
AU - Eugene Vinitsky
AU - Yonathan Efroni
PY - 2026
DA - 2026/09/01
PB - arXiv
DO - 10.48550/arxiv.2609.02928
UR - https://arxiv.org/abs/2609.02928
AB - Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.
ER -