Tail-Likelihood Reinforcement Learning

Abstract

Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.

Separating Weight and Likelihood Tail Effects in Bayesian Neural Groundwater Inversion
Ke-Ming Shen
arXiv1 Nov 2026PreprintCC BY-NC-ND 4.0Full textGeophysics & seismology StatisticsarXiv
Pathwise-Budgeted LLM Routing: Online Allocation with K-Bit Tail-Cost Learning
Hanzhang Qin, Sophie H. Yu, Yuewei Zhang +1
arXiv4 Nov 2026PreprintarXiv licenceFull textOptimisation & controlarXiv
Reinforcement Learning Under Runtime Confounding: Policy Learning and Off-Policy Evaluation
Raphael Kim, Michele Santacatterina, Ramin Zabih +1
arXiv29 Oct 2026PreprintCC BY 4.0Full textStatisticsarXiv
Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems
Oumaïma Bounhar, Thomas Chêne, Ghaya Rekaya-Ben Othman +1
arXiv29 Oct 2026PreprintCC BY-NC-SA 4.0Full textStatisticsarXiv
Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
Zeru Zhu, Marco Gamarra, Ji Liu
arXiv8 Oct 2026PreprintarXiv licenceFull textCombinatorics & discrete mathematics Optimisation & controlarXiv

Cite

BibTeX RIS

BibTeX

@misc{ramasubramanian2026taillikelihood,
  title = {{Tail-Likelihood Reinforcement Learning}},
  author = {Shrinivas Ramasubramanian and Daman Arora and Fahim Tajwar and Guanning Zeng and Qingyang Wu and Zhongzhu Zhou and Chenfeng Xu and Haiwen Feng and Yuda Song and Aarti Singh and Ruslan Salakhutdinov and J. Andrew Bagnell and Jeff Schneider and Andrea Zanette},
  howpublished = {arXiv},
  year = {2026},
  month = sep,
  doi = {10.48550/arxiv.2609.02987},
  eprint = {2609.02987},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.02987},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Tail-Likelihood Reinforcement Learning
AU  - Shrinivas Ramasubramanian
AU  - Daman Arora
AU  - Fahim Tajwar
AU  - Guanning Zeng
AU  - Qingyang Wu
AU  - Zhongzhu Zhou
AU  - Chenfeng Xu
AU  - Haiwen Feng
AU  - Yuda Song
AU  - Aarti Singh
AU  - Ruslan Salakhutdinov
AU  - J. Andrew Bagnell
AU  - Jeff Schneider
AU  - Andrea Zanette
PY  - 2026
DA  - 2026/09/09
PB  - arXiv
DO  - 10.48550/arxiv.2609.02987
UR  - https://arxiv.org/abs/2609.02987
AB  - Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.
ER  -

Abstract

Related

Cite