BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning
Abstract
Agentic reinforcement learning (ARL) with verifiable rewards improves the ability of large language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and reasoning. However, most existing ARL methods optimize only LLM-generated tokens and treat retrieved evidence as environment observations. This creates an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than to the retriever, which motivates training the LLM and the retriever jointly. In this paper, we show that retrieval and LLM policy learning are order-sensitive: adapting the retriever before optimizing the policy yields a larger reward gain than the reverse order. To preserve this hierarchy while allowing both components to co-adapt, we formulate retrieval-augmented agentic RL as a bilevel optimization problem. To solve it efficiently, we introduce BRIDGE, a memory-efficient first-order bilevel method motivated by a loss-landscape analysis of the RL and retrieval objectives. Across seven open-domain QA benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving the multi-hop average over the strongest baseline by 9.6 and 3.4 EM points, respectively. It also achieves the best averaged answer accuracy and reasoning quality across medical QA benchmarks.
Related
- Back to the Definition: Estimating Step-Level Advantages via Trajectory Graphs for Agentic Reinforcement Learning
- Bi-ZOL: Bilevel Zeroth-Order Learning with Nonsmooth Responses
- Bilevel optimization for data-driven learning of Koopman embeddings using kernel-based autoencoders
- DagFlow AI: Agentic Synthetic Data Generation for Personalized Statistics Learning
- Retrieval-Based In-Context Learning: A Domain Adaptation Framework
BibTeX
@misc{xiao2026bridge,
title = {{BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning}},
author = {Quan Xiao and Mingda Liu and Gaowen Liu and Katsuki Fujisawa and Tianyi Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.36505},
eprint = {2609.36505},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.36505},
}
RIS
TY - UNPB
TI - BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning
AU - Quan Xiao
AU - Mingda Liu
AU - Gaowen Liu
AU - Katsuki Fujisawa
AU - Tianyi Chen
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2609.36505
UR - https://arxiv.org/abs/2609.36505
AB - Agentic reinforcement learning (ARL) with verifiable rewards improves the ability of large language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and reasoning. However, most existing ARL methods optimize only LLM-generated tokens and treat retrieved evidence as environment observations. This creates an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than to the retriever, which motivates training the LLM and the retriever jointly. In this paper, we show that retrieval and LLM policy learning are order-sensitive: adapting the retriever before optimizing the policy yields a larger reward gain than the reverse order. To preserve this hierarchy while allowing both components to co-adapt, we formulate retrieval-augmented agentic RL as a bilevel optimization problem. To solve it efficiently, we introduce BRIDGE, a memory-efficient first-order bilevel method motivated by a loss-landscape analysis of the RL and retrieval objectives. Across seven open-domain QA benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving the multi-hop average over the strongest baseline by 9.6 and 3.4 EM points, respectively. It also achieves the best averaged answer accuracy and reasoning quality across medical QA benchmarks.
ER -