← Back to all news
arXiv Machine Learning By Taoran Liang, Yang Liu, Shang Luo, Yingguang Yang, Rongrong Zhang, Yingzong Min, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Kefu Xu, Congjing Ran, Bin Chong

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

Read the original on arXiv Machine Learning →

The paper introduces GACA, a critic‑free reinforcement learning estimator that adapts credit assignment granularity based on a step‑level uncertainty proxy. GACA assigns higher weight to fine‑grained signals for steps with above‑average negative log‑likelihood, while relying on episode‑level signals for less uncertain steps, improving task success on ALFWorld and WebShop for 1.5B and 7B language models. The authors provide a risk decomposition, a conditional bound on action‑value variation, and an error‑projection analysis to justify the method’s effectiveness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 5

AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning

AdaStep introduces an adaptive step-credit weighting technique for agentic reinforcement learning, addressing the coarse granularity of trajectory-level objectives in long-horizon LLM agents. By formulating the weighting as a mean-squared-error estimation problem and deriving an optimal per-state shrinkage coefficient, AdaStep selectively preserves local credit when return variation is due to the chosen action and suppresses it when downstream randomness dominates. The method requires only lightweight scalar computations, no critic or extra rollouts, and demonstrates consistent performance gains across three model backbones on ALFWorld, WebShop, and ScienceWorld.

By Xin Wang, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, Jian Luan
llmsagentsreinforcement-learning
More like this →
arXiv AI
Oct 19

Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.

By Yunan Wang, Minghui Song, Zihan Zhang, Shaohan Huang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
llmsagentsreinforcement-learningefficiencybenchmarkssafety
More like this →