arXiv AI

Learning from Viable Failure Prefixes: Milestone Viability Potential Policy Optimization for Long-Horizon LLM Agents

The paper introduces Milestone Viability Potential Policy Optimization (MVPO), a reinforcement learning algorithm designed for long‑horizon large language model agents. MVPO addresses the zero‑credit failure problem by learning from viable failure prefixes, estimating prefix potential over Union‑Find viability regions, and repairing zero‑credit groups with potential‑difference advantages. Experiments on Qwen2.5‑1.5B‑Instruct demonstrate that MVPO outperforms eight strong baselines, improving success rates on ALFWorld and WebShop with minimal overhead.

arXiv AI

Sep 3

PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.

By Yuyao Zheng, Haipeng Sun, Junwei Bao, Lemao Liu, Hongfei Jiang, Yang Song, Dejing Dou

llms agents reinforcement-learning

arXiv AI

Aug 21

MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents

arXiv:2608. 19803v1 Announce Type: cross Abstract: Credit assignment is challenging in long-horizon agentic reinforcement learning, where supervision often comes only from final rewards.

By Bo Qian, Yuting Wu, Shuang Zeng, Huaiyu Wan, Dalin Zhang, Jiqiang Liu

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 5

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

arXiv:2606. 05885v1 Announce Type: new Abstract: Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards.

By Yuanfan Li, Qi Zhou, Wenjing Duan, Lu Chen

llms agents reinforcement-learning safety

Hugging Face Trending Papers

Sep 2

PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks proposes a new reinforcement learning approach that estimates empirical state potentials from anchor-state-group return statistics within each rollout group. By deriving action advantages from potential differences between adjacent states, PGPO enables cross‑trajectory credit propagation, providing finer‑grained step‑level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong overall performance compared to recent group‑based RL methods, with negligible training overhead.

llms agents reinforcement-learning

Hugging Face Trending Papers

Jun 24

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed.

llms agents reinforcement-learning

arXiv AI

4d ago

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

arXiv:2609.36178v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...

By Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li, Yuexing Hao, Yu Hu, Muhao Chen, Varun Chandrasekaran, Andrzej Banburski-Fahey, Jaron Lanier

llms agents reinforcement-learning

arXiv Machine Learning

Jun 25

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

arXiv:2606. 25852v1 Announce Type: new Abstract: Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes.

By Peng Xu, Sijia Chen, Junzhuo Li, Xuming Hu

llms agents reinforcement-learning

arXiv AI

Jul 28

Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks

arXiv:2607. 22724v1 Announce Type: cross Abstract: Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group.

By Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng

llms agents benchmarks

Hugging Face Trending Papers

Aug 17

TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents

Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult. Existing approache...

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 15

GAGPO: Generalized Advantage Grouped Policy Optimization

arXiv:2605. 13217v1 Announce Type: cross Abstract: Reinforcement learning has become a powerful paradigm for post-training large language model agents, yet credit assignment in multi-turn environments remains a challenge.

By Siyuan Zhu, Chao Yu, Rongxin Yang, Zongkai Liu, Jinjun Hu, Qiwen Chen, Yibo Zhang

llms agents reinforcement-learning

arXiv AI

2d ago

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

arXiv:2610.00838v1 Announce Type: cross Abstract: Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error...

By Xinchen Du, Zhengze Zhou, Wenhui Zhu, Han Yu, Sen Na, Rohit Jain, Alborz Geramifard

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 7

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

arXiv:2607. 04242v1 Announce Type: new Abstract: Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks.

By Mingxuan Fan, Peiyang Liu

llms agents reinforcement-learning

Related stories