KnowGPO introduces a reinforcement learning framework that groups search agent trajectories by the task-relevant knowledge they acquire, rather than by surface observations. By decomposing required information into atomic knowledge conditions and using the incremental knowledge gain as a process reward, KnowGPO provides finer-grained credit assignment for deep search tasks. Experiments on Xbench, BrowseComp-ZH, and GAIA demonstrate that KnowGPO outperforms outcome-only and process-supervision baselines by 3.8 to 4.2 points, with an 8B model rivaling or exceeding most 32B open-source systems.
By Siqi Chen, Zhengxi Lu, Yuchen Yan, Xingyu Wu, Qipeng Chen, Xin Zhang, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Weiming Lu, Jun Xiao, Yongliang Shen
arXiv:2605. 17877v2 Announce Type: replace Abstract: A significant hurdle for current LLMs is the execution of complex, multi-stage tasks.
By Wonjoong Kim, Yeonjun In, Sangwu Park, Dongha Lee, Chanyoung Park
arXiv:2610.00388v1 Announce Type: cross
Abstract: Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However,...
By Bo-Wen Zhang, Junwei He, Maoqi Liu, Feiran Li, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo
arXiv:2608.21946v1 Announce Type: cross
Abstract: Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exp...
By Can Xie, Yuyi Zhou, Wen Yang, Ziyi zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, Jiajun Zhang
arXiv:2610.18614v1 Announce Type: new
Abstract: Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human....
By Ashutosh Hathidara
MOER is a mix‑policy optimization framework that blends expert guidance with self‑exploration in reinforcement learning. It injects a decaying expert chain‑of‑thought prefix into prompts, progressively annealing the prefix length to encourage exploration, and uses a selective rollback mechanism to replace failed completions with the full expert trace, ensuring correctness. The method employs a harmonized mix‑policy objective that optimizes on‑policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust‑region SFT loss, leading to stable training and superior performance on mathematical and general reasoning tasks.
By Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu
Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.
By Yunan Wang, Minghui Song, Zihan Zhang, Shaohan Huang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
arXiv:2607. 21419v1 Announce Type: new Abstract: In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization.
By Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, Zhengzhou Zhu
arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.
By Yu Han, Kailing Li, Yang Jiao, Yulin Dai, Yuqian Fu, Linhai Zhuo, Tianwen Qian
The paper introduces a steering agent that supervises large language model (LLM)-based agents and multi‑agent systems during execution to prevent goal drift. By observing the execution context, the steering agent decides when to intervene using either a low‑cost fast path or a high‑cost slow path, and both paths are jointly trained with reinforcement learning. Experiments on information‑seeking search tasks demonstrate that the steering agent improves performance and efficiency for both single‑agent and multi‑agent systems, generalizes to out‑of‑domain tasks, and allows controllable intervention frequency without retraining.
By Leyang Shen, Yaxin Gong, Yang Zhang, Chenxiao Fan, Jiakai Wang, Dong Wang, Yang Liu, Fuli Feng, Chun Kai Ling, Tat-Seng Chua
arXiv:2607. 15610v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation.
By Xintong Li, Sha Li, Yuwei Zhang, Changlong Yu, Rongmei Lin, Hongye Jin, Shuyi Guan, Xin Liu, Linwei Li, Qingyu Yin, Jingbo Shang
The paper introduces Rollout-Time Intervention (RTI), a method that uses reward-side evidence during rollouts to transform unsuccessful trajectories into successful ones, thereby providing learning signals for tasks beyond a policy’s intrinsic exploration limits. RTI filters out trajectories with excessive distributional shifts, allowing these off-policy interventions to guide policy optimization effectively. Experiments on five benchmarks show that RTI consistently outperforms existing RL baselines, enabling post-trained policies to approach the performance of stronger large language models and surpass their original capability boundaries.
By Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui