AHEAD is a step‑aware framework that augments reinforcement learning for multi‑turn LLM agents by matching different supervision sources to different step types. The teacher receives environment feedback on all steps and LLM‑generated corrective hints only on error steps, providing finer‑grained guidance than uniform trajectory‑level rewards. Across ALFWorld, WebShop, and Search‑based QA, AHEAD improves task success by 13.3 points on ALFWorld and 11.0 on WebShop at 7B, reaches target success rates faster, and solves tasks within tighter interaction budgets compared to outcome‑only RL and prior self‑distillation baselines.
By Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar
arXiv:2606. 11559v1 Announce Type: new Abstract: Reinforcement learning typically improves multi-turn agent capabilities through the terminal outcome of the trajectories, which makes it difficult to determine credit assignments for each intermediate turns.
By Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, Jingbo Shang
arXiv:2611.05266v2 Announce Type: replace
Abstract: Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from c...
By Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair
arXiv:2608. 07371v1 Announce Type: new Abstract: Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards.
By Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang
The paper introduces Self‑Retrospection Distillation (SRD), a method that uses post‑hoc trajectories to supervise foresight predictions made before an agent acts. SRD distills information from completed rollouts into trajectory‑blind predictions, enabling the agent to anticipate useful actions and avoid pitfalls without requiring explicit foresight at inference time. Experiments on ten tool‑integrated reasoning and long‑horizon tasks show that SRD improves performance by up to 24.2 percentage points, especially when reward signals are scarce or uniform across rollouts.
By Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
SCOPE is a new framework for selective critical‑step on‑policy self‑distillation in agentic reinforcement learning. After a supervised warm‑up, the policy examines its own trajectories, generates hindsight skills for each step, and distills guidance only at selected error steps, using a bounded teacher‑student coefficient and a success‑relative pairwise length reward. Experiments on ALFWorld and WebShop show consistent performance gains across multiple model backbones, with Qwen2.5‑3B achieving 97.7 % success on ALFWorld and 85.2 % on WebShop, and Qwen3‑32B reaching 99.2 % on seen tasks and 94.0 % on unseen tasks.
By Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu
arXiv:2606. 18890v1 Announce Type: new Abstract: Improving GUI agents typically relies on behavior cloning on expert trajectories.
By Zhimin Fan, Hongwei Yu, Yeqing Shen, Haolong Yan, Guozhen Peng, Tianhao Peng, Yudong Zhang, Xiaowen Zhang, Kaijun Tan, Zheng Ge, Xiangyu Zhang, Daxin Jiang
arXiv:2608. 04007v1 Announce Type: cross Abstract: Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions.
By Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou, Xinran Chen, Simon, Jun Xu
arXiv:2607. 28076v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents.
By Binbin Zheng, Zijun Xie, Guanqun Zhao, Enlei Gong, Xing Ma, Xiaoliang Fu, Zeyu Chen
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.
By Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
Prospective Hindsight (PH) is a self‑calibrating training principle for reinforcement learning that augments any retrospective base method with a signal derived from the gap between an agent’s prospective prediction and its retrospective evaluation. By weighting gradients with a stop‑gradient surprise based on this prediction‑reality gap, PH highlights samples where the agent’s self‑model is most inaccurate and encourages the policy and predictor to co‑evolve, progressively focusing on remaining blind spots. Experiments on single‑turn verifiable tasks and a multi‑turn personal‑agent task show that PH improves both task performance and calibration across model scales, addressing overconfident failures in single‑turn settings and underconfident successes in multi‑turn settings.
By Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Yu Li, Hiroaki Hayashi, Chien-Sheng Wu
arXiv:2607. 05541v1 Announce Type: cross Abstract: Reinforcement Learning is commonly used to train large language models using environmental feedback.
By Muhammad Zain Amin, Kibele Sebnem Yildirim