The paper introduces FACA, a Feedback‑Aware Credit Assignment method that aligns each agent reaction with the preceding user‑to‑user segment, derives a locally normalized reaction advantage, and adds it to the terminal outcome advantage without requiring an extra critic or rollout. Compared to an outcome‑only Interactive GRPO baseline, FACA improves performance across nine domains by 5.91–10.22 percentage points on 8B and 14B models, with notable gains in Telecom. The approach demonstrates that next‑turn user reactions provide actionable local credit for enhancing multi‑turn user‑interacting agents.
arXiv:2603. 06194v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment.
By Naifan Zhang, Ruihan Sun, Jinwei Su, Hengjie Yang, Zhengyuan Pan, Zhaohan Chen, Xiaofan Zhang
arXiv:2602. 11351v2 Announce Type: replace Abstract: Proactive large language model (LLM) agents aim to actively plan, query, and interact over multiple turns, enabling efficient task completion beyond passive instruction following and making them essential for real-world, user-centric applications.
By Yihang Yao, Zhepeng Cen, Haohong Lin, Shiqi Liu, Zuxin Liu, Jiacheng Zhu, Zhang-Wei Hong, Laixi Shi, Ding Zhao
Iterative GRPO is a batch‑online policy iteration framework that enables multi‑turn reinforcement learning for conversational agents without requiring an interactive user simulator. It alternates between learning a turn‑level Q‑function from logged returns (policy evaluation) and applying single‑turn GRPO against this Q‑function (policy improvement), thereby scoring candidate responses by their expected downstream return. The method is validated on six multi‑turn negotiation environments, demonstrating its practicality for real‑world deployment patterns.
By Daniel R. Jiang, Ankur Samanta, Yukai Yang, Jalaj Bhandari, R\'emi Munos, Tyler Lu
arXiv:2511.21638v3 Announce Type: replace
Abstract: Practical LLM agents often operate over multi-turn conversations where success is determined only after the full interaction ends. Most multi-turn...
By Daniel R. Jiang, Ankur Samanta, Yukai Yang, Jalaj Bhandari, R\'emi Munos, Tyler Lu
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.
By Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu