The paper introduces Multiscale Policy Optimization (MPO) for long‑horizon large language model agents, addressing the mismatch between turn‑level critic values and token‑level action generation. MPO combines Multiscale Advantage Estimation (MAE) with turn‑level importance sampling to better align credit assignment and policy updates at the turn scale. Experiments on search, tool‑use, and automation tasks with 4B and 30B‑A3B models show consistent improvements over strong reinforcement learning baselines.
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motiva...
The paper introduces CrEST, a hierarchical credit assignment framework for multi-turn, multi-step large language model agents. CrEST anchors each token’s advantage sign to verifier-defined rewards while using a self-teacher to provide dense magnitude signals, addressing both inter-turn dilution and intra-turn token contribution issues. Experiments on BFCL V3 and WildToolBench show CrEST consistently outperforms reinforcement learning and distillation baselines, especially on long trajectories and strict session-level metrics.
Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.
The paper introduces Effect-Marginal Policy Optimization (EMPO), a method that addresses credit-update misalignment in multi-turn reinforcement learning for large language model agents. EMPO aligns policy updates with execution-level equivalence by using an effect-level surrogate that couples updates across different textual responses that produce the same effect. Experiments on ALFWorld, WebShop, and Sokoban demonstrate that EMPO improves both task performance and learning efficiency compared to strong baselines.
arXiv:2610.08402v1 Announce Type: new Abstract: Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two r...
Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over t...
The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.
The paper introduces Influence-Aware Policy Optimization (IAPO), a method that models multi‑turn agent rollouts as typed influence‑dependency graphs to better assign credit to actions based on how information and errors flow through user and tool interactions. IAPO transforms the structure of support and failure usage into routing weights that redistribute trajectory‑level advantage, enabling more effective learning from sparse final rewards. Experiments with Qwen3‑4B and Qwen3‑8B on three service‑agent benchmarks show that IAPO outperforms existing multi‑turn reinforcement learning baselines without harming function‑calling performance.
Reinforcement learning (RL) has emerged as the primary paradigm for optimizing Large Language Model (LLM) agents, yet efficient critic-free algorithms (e.g., GRPO) suffer from a fundamental uniform cr...
arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.
The paper introduces GRAFT, a Graph-based Faithful sTep-level credit-assignment framework that constructs a trajectory graph from rollout trajectories, recovers node state-values via Bellman iteration, and assigns step-level advantages based on node value differences. It also proposes Graph GAE to further reduce state-value estimation bias. Experiments on multi-turn agentic benchmarks demonstrate consistent improvements over GRPO and other recent agentic RL algorithms.
The paper introduces ProAS, a Progress‑Aware Advantage Shaping framework designed to improve critic‑free reinforcement learning for large language model agents. ProAS uses a dynamic Agentic Reference Graph to provide fine‑grained credit assignment through Masking Unreasonable Incentives and Asymmetric Penalty Redistribution, avoiding the uniform advantage broadcast that can reinforce invalid actions. Experiments on Qwen2.5‑1.5B‑Instruct and Qwen2.5‑7B‑Instruct show that ProAS raises baseline success rates to 91.9% and 94.3% on ALFWorld and 66.7% and 70.8% on WebShop, while speeding up training convergence up to 2.44× and remaining robust to imperfect reference graphs.