Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.
By Yunan Wang, Minghui Song, Zihan Zhang, Shaohan Huang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
arXiv:2607. 22724v1 Announce Type: cross Abstract: Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group.
By Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng
arXiv:2608. 19842v1 Announce Type: new Abstract: Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models.
By Dayang Liang, Lang Feng, Bo An, Yunlong Liu
arXiv:2608. 19803v1 Announce Type: cross Abstract: Credit assignment is challenging in long-horizon agentic reinforcement learning, where supervision often comes only from final rewards.
By Bo Qian, Yuting Wu, Shuang Zeng, Huaiyu Wan, Dalin Zhang, Jiqiang Liu
The paper introduces Long-Short Term Advantage Estimator (LSTAE), a single‑stream reinforcement learning algorithm that replaces repeated within‑iteration trajectory sampling with historical experience for advantage estimation. LSTAE tracks each task anchor with a drift‑aware long‑term baseline and a short‑term state‑experience buffer, converting accumulated experience into multi‑granular credit signals while requiring only one rollout per anchor. Experiments on agentic and mathematical reasoning benchmarks show that LSTAE matches or surpasses strong group‑based baselines while significantly reducing rollout cost.
By Xinhao Yao, Lu Yu, Changhao Wang, Fengwei Teng, Yuyao Zhang, Qing Cui, Jun Zhou, Yong Liu
arXiv:2610.07349v1 Announce Type: cross
Abstract: Group Relative Policy Optimization (GRPO) avoids a separate critic by estimating advantages from rollout groups. For multi-turn agents, however, traj...
By Sayak Chakrabarti, Sathish Reddy Indurthi