arXiv:2606. 05885v1 Announce Type: new Abstract: Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards.
By Yuanfan Li, Qi Zhou, Wenjing Duan, Lu Chen
The paper introduces GACA, a critic‑free reinforcement learning estimator that adapts credit assignment granularity based on a step‑level uncertainty proxy. GACA assigns higher weight to fine‑grained signals for steps with above‑average negative log‑likelihood, while relying on episode‑level signals for less uncertain steps, improving task success on ALFWorld and WebShop for 1.5B and 7B language models. The authors provide a risk decomposition, a conditional bound on action‑value variation, and an error‑projection analysis to justify the method’s effectiveness.
By Taoran Liang, Yang Liu, Shang Luo, Yingguang Yang, Rongrong Zhang, Yingzong Min, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Kefu Xu, Congjing Ran, Bin Chong
arXiv:2608. 06861v1 Announce Type: new Abstract: Training large language model agents in long-horizon environments requires assigning credit from sparse terminal outcomes to individual actions.
By Hongxi Yan, Ziyue Huang, Shichao Fan, Qingjie Liu
The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.
By Yuyao Zheng, Haipeng Sun, Junwei Bao, Lemao Liu, Hongfei Jiang, Yang Song, Dejing Dou
arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.
By Yu Han, Kailing Li, Yang Jiao, Yulin Dai, Yuqian Fu, Linhai Zhuo, Tianwen Qian
The paper introduces GRAFT, a Graph-based Faithful sTep-level credit-assignment framework that constructs a trajectory graph from rollout trajectories, recovers node state-values via Bellman iteration, and assigns step-level advantages based on node value differences. It also proposes Graph GAE to further reduce state-value estimation bias. Experiments on multi-turn agentic benchmarks demonstrate consistent improvements over GRPO and other recent agentic RL algorithms.
By Xincheng Yao, Haobo Fu, Weiming Liu, Chongyang Zhang