The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper introduces GRAFT, a Graph-based Faithful sTep-level credit-assignment framework that constructs a trajectory graph from rollout trajectories, recovers node state-values via Bellman iteration, and assigns step-level advantages based on node value differences. It also proposes Graph GAE to further reduce state-value estimation bias. Experiments on multi-turn agentic benchmarks demonstrate consistent improvements over GRPO and other recent agentic RL algorithms.
The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.
arXiv:2607. 15610v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation.
PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks proposes a new reinforcement learning approach that estimates empirical state potentials from anchor-state-group return statistics within each rollout group. By deriving action advantages from potential differences between adjacent states, PGPO enables cross‑trajectory credit propagation, providing finer‑grained step‑level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong overall performance compared to recent group‑based RL methods, with negligible training overhead.
arXiv:2605. 26684v2 Announce Type: replace-cross Abstract: Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks.
Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks.