AHEAD is a step‑aware framework that augments reinforcement learning for multi‑turn LLM agents by matching different supervision sources to different step types. The teacher receives environment feedback on all steps and LLM‑generated corrective hints only on error steps, providing finer‑grained guidance than uniform trajectory‑level rewards. Across ALFWorld, WebShop, and Search‑based QA, AHEAD improves task success by 13.3 points on ALFWorld and 11.0 on WebShop at 7B, reaches target success rates faster, and solves tasks within tighter interaction budgets compared to outcome‑only RL and prior self‑distillation baselines.
By Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar
arXiv:2606. 11559v1 Announce Type: new Abstract: Reinforcement learning typically improves multi-turn agent capabilities through the terminal outcome of the trajectories, which makes it difficult to determine credit assignments for each intermediate turns.
By Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, Jingbo Shang
arXiv:2611.05266v2 Announce Type: replace
Abstract: Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from c...
By Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair
arXiv:2608. 07371v1 Announce Type: new Abstract: Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards.
By Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang
The paper introduces Self‑Retrospection Distillation (SRD), a method that uses post‑hoc trajectories to supervise foresight predictions made before an agent acts. SRD distills information from completed rollouts into trajectory‑blind predictions, enabling the agent to anticipate useful actions and avoid pitfalls without requiring explicit foresight at inference time. Experiments on ten tool‑integrated reasoning and long‑horizon tasks show that SRD improves performance by up to 24.2 percentage points, especially when reward signals are scarce or uniform across rollouts.
By Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu
SCOPE is a new framework for selective critical‑step on‑policy self‑distillation in agentic reinforcement learning. After a supervised warm‑up, the policy examines its own trajectories, generates hindsight skills for each step, and distills guidance only at selected error steps, using a bounded teacher‑student coefficient and a success‑relative pairwise length reward. Experiments on ALFWorld and WebShop show consistent performance gains across multiple model backbones, with Qwen2.5‑3B achieving 97.7 % success on ALFWorld and 85.2 % on WebShop, and Qwen3‑32B reaching 99.2 % on seen tasks and 94.0 % on unseen tasks.
By Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu