Hugging Face Trending Papers
Multiscale Policy Optimization for Long-Horizon LLM Agents
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
1d ago
Multiscale Policy Optimization for Long-Horizon LLM Agents
The paper introduces Multiscale Policy Optimization (MPO) for long‑horizon large language model agents, addressing the mismatch between turn‑level critic values and token‑level action generation. MPO combines Multiscale Advantage Estimation (MAE) with turn‑level importance sampling to better align credit assignment and policy updates at the turn scale. Experiments on search, tool‑use, and automation tasks with 4B and 30B‑A3B models show consistent improvements over strong reinforcement learning baselines.
arXiv AI
Oct 19
Teach the Magnitude, Not the Direction: Verifier-Anchored Credit Assignment for Multi-Turn Multi-Step LLM Agents
The paper introduces CrEST, a hierarchical credit assignment framework for multi-turn, multi-step large language model agents. CrEST anchors each token’s advantage sign to verifier-defined rewards while using a self-teacher to provide dense magnitude signals, addressing both inter-turn dilution and intra-turn token contribution issues. Experiments on BFCL V3 and WildToolBench show CrEST consistently outperforms reinforcement learning and distillation baselines, especially on long trajectories and strict session-level metrics.
Hugging Face Trending Papers
Aug 25
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over t...
arXiv AI
Oct 19
Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.
arXiv Machine Learning
6d ago
One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents
The paper introduces Effect-Marginal Policy Optimization (EMPO), a method that addresses credit-update misalignment in multi-turn reinforcement learning for large language model agents. EMPO aligns policy updates with execution-level equivalence by using an effect-level surrogate that couples updates across different textual responses that produce the same effect. Experiments on ALFWorld, WebShop, and Sokoban demonstrate that EMPO improves both task performance and learning efficiency compared to strong baselines.
arXiv AI
Sep 3
PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
The paper introduces Potential-Guided Policy Optimization (PGPO), a method for multi-turn agentic tasks that improves credit assignment by estimating empirical state potentials from anchor-state-group return statistics. PGPO derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation and finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop demonstrate strong performance compared to recent group-based reinforcement learning methods, with negligible training overhead.