A learner-free swarm reaches the explore-exploit equilibrium: A self-organized attractor approximating the reinforcement-learning solution
Abstract
The explore-exploit trade-off is the core problem of reinforcement learning (RL). In its multi-agent, resource-sharing form the optimum is not a scalar-return maximum but an equilibrium allocation -- the ideal-free distribution (IFD), the Nash equilibrium of a congestion game. We show that a minimal foraging swarm reaches the shape of this equilibrium as a self-organized attractor -- foragers in proportion to resource richness (matching slope 0.94) -- with no learner, no trained parameters, and no payoff or reward read anywhere in its dynamics. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; removing the coupling of recruitment to movement collapses the matching (slope 0.08). The reinforcement learning is at the level of the problem and the allocation reached, not the micro-rule. The swarm reaches this equilibrium rather than learning it, attaining its shape but not the exact equilibrium: it stays measurably exploitable (Nash-gap 0.58 versus ~0.01 for a converged learner). Against PPO learners (feedforward, centralized-critic, and graph message-passing) trained in the same sensing-limited environment, the reward-free swarm at zero training matches the IFD gradient more faithfully than any learner (slope ~1 vs 0.69-0.91) and ties the best learner's harvest where clustered (both plateau at ~0.80 of an oracle ceiling), while the learner collects more where resource is scarce (0.33 vs 0.22) and holds tighter per-capita intake. Learning thus buys precision and transfer, not a better allocation; neither message passing nor a centralized critic exceeds it at this budget. We read this not as beating RL, but as evidence that a minimal swarm's physics reaches the equilibrium RL seeks without a learner, a reward, or a trained parameter. A companion paper develops the mechanism.
Related
- Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
- Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
- Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
- Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment
- Equilibrium Learning in Unbounded Stochastic Games with Gaussian Transitions
BibTeX
@misc{wilkerson2026learnerfree,
title = {{A learner-free swarm reaches the explore-exploit equilibrium: A self-organized attractor approximating the reinforcement-learning solution}},
author = {Galen J. Wilkerson},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.25113},
eprint = {2610.25113},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.25113},
}
RIS
TY - UNPB
TI - A learner-free swarm reaches the explore-exploit equilibrium: A self-organized attractor approximating the reinforcement-learning solution
AU - Galen J. Wilkerson
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.25113
UR - https://arxiv.org/abs/2610.25113
AB - The explore-exploit trade-off is the core problem of reinforcement learning (RL). In its multi-agent, resource-sharing form the optimum is not a scalar-return maximum but an equilibrium allocation -- the ideal-free distribution (IFD), the Nash equilibrium of a congestion game. We show that a minimal foraging swarm reaches the shape of this equilibrium as a self-organized attractor -- foragers in proportion to resource richness (matching slope 0.94) -- with no learner, no trained parameters, and no payoff or reward read anywhere in its dynamics. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; removing the coupling of recruitment to movement collapses the matching (slope 0.08). The reinforcement learning is at the level of the problem and the allocation reached, not the micro-rule. The swarm reaches this equilibrium rather than learning it, attaining its shape but not the exact equilibrium: it stays measurably exploitable (Nash-gap 0.58 versus ~0.01 for a converged learner). Against PPO learners (feedforward, centralized-critic, and graph message-passing) trained in the same sensing-limited environment, the reward-free swarm at zero training matches the IFD gradient more faithfully than any learner (slope ~1 vs 0.69-0.91) and ties the best learner's harvest where clustered (both plateau at ~0.80 of an oracle ceiling), while the learner collects more where resource is scarce (0.33 vs 0.22) and holds tighter per-capita intake. Learning thus buys precision and transfer, not a better allocation; neither message passing nor a centralized critic exceeds it at this budget. We read this not as beating RL, but as evidence that a minimal swarm's physics reaches the equilibrium RL seeks without a learner, a reward, or a trained parameter. A companion paper develops the mechanism.
ER -