The paper introduces Rollout-Time Intervention (RTI), a method that uses reward-side evidence during rollouts to transform unsuccessful trajectories into successful ones, thereby providing learning signals for tasks beyond a policy’s intrinsic exploration limits. RTI filters out trajectories with excessive distributional shifts, allowing these off-policy interventions to guide policy optimization effectively. Experiments on five benchmarks show that RTI consistently outperforms existing RL baselines, enabling post-trained policies to approach the performance of stronger large language models and surpass their original capability boundaries.
By Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui
arXiv:2606. 17680v1 Announce Type: new Abstract: Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents.
By Zhitong Wang, Songze Li, Hao Peng, Shuzheng Si, Yi Wang, Maosong Sun, Juanzi Li
Large language model (LLM) agents for long-horizon interactive tasks typically follow a ReAct-style protocol, issuing one primitive action per LLM round. While this enables frequent replanning, it is...
The paper introduces SPACE, a method for enabling large language model agents to emit variable-length action chunks in long-horizon tasks. By distilling chunk-boundary supervision from programmatic skills derived from successful trajectories, SPACE overcomes the tendency of agents to either act one step at a time or commit to overly long sequences. Experiments on ALFWorld and ScienceWorld demonstrate that SPACE raises success rates by 7.0%–31.3% and cuts LLM decision rounds by up to 78.9%.
By Yanting Yang, Can Jin, Jinman Zhao, Jiahao Wu, Yang Zhou, Zhepeng Wang, Zhendong Wang, Mu Zhou, Dimitris N. Metaxas
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.
By Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu
arXiv:2601. 15141v2 Announce Type: replace Abstract: Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving.
By Tianshi Xu, Yuteng Chen, Meng Li