arXiv:2606. 25852v1 Announce Type: new Abstract: Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes.
By Peng Xu, Sijia Chen, Junzhuo Li, Xuming Hu
The paper introduces Milestone Viability Potential Policy Optimization (MVPO), a reinforcement learning algorithm designed for long‑horizon large language model agents. MVPO addresses the zero‑credit failure problem by learning from viable failure prefixes, estimating prefix potential over Union‑Find viability regions, and repairing zero‑credit groups with potential‑difference advantages. Experiments on Qwen2.5‑1.5B‑Instruct demonstrate that MVPO outperforms eight strong baselines, improving success rates on ALFWorld and WebShop with minimal overhead.
By Qi Zhou, Yuanfan Li
The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.
By Yuyao Zheng, Haipeng Sun, Junwei Bao, Lemao Liu, Hongfei Jiang, Yang Song, Dejing Dou
Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult. Existing approache...
PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks proposes a new reinforcement learning approach that estimates empirical state potentials from anchor-state-group return statistics within each rollout group. By deriving action advantages from potential differences between adjacent states, PGPO enables cross‑trajectory credit propagation, providing finer‑grained step‑level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong overall performance compared to recent group‑based RL methods, with negligible training overhead.
arXiv:2608. 19803v1 Announce Type: cross Abstract: Credit assignment is challenging in long-horizon agentic reinforcement learning, where supervision often comes only from final rewards.
By Bo Qian, Yuting Wu, Shuang Zeng, Huaiyu Wan, Dalin Zhang, Jiqiang Liu