arXiv Machine Learning
What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 20
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
The paper introduces Reverse‑Turn Policy Optimization (RTPO), a method that restructures multi‑turn agentic reinforcement learning rollouts into sparse reverse trees and updates policies in temporal reverse order. This approach addresses three key instability sources—context mismatch, weak turn‑level credit assignment, and asynchronous policy drift—by aligning each decision with its downstream continuation. Theoretical analysis shows RTPO eliminates context mismatch and drift, reduces credit bias, and converges to recursive optimality, while experiments demonstrate performance gains of 21.50% over trajectory‑level and 10.76% over turn‑level baselines on multi‑turn agentic RL benchmarks.
arXiv AI
Jul 20
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
arXiv:2607. 15610v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation.
arXiv AI
Jun 2
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
arXiv:2606. 00135v1 Announce Type: cross Abstract: Tool-calling is a central component of modern large language model (LLM) agents, equipping them with skills beyond their parametric knowledge.
arXiv Machine Learning
Jun 25
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
arXiv:2606. 26027v1 Announce Type: cross Abstract: Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities.
arXiv AI
Jun 16
PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents
arXiv:2606. 16215v1 Announce Type: cross Abstract: Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns.
arXiv AI
Aug 19
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.