arXiv AI October 7, 2026 By Chenliang Li, Adel Elmahdy, Alex Boyd, Zhongruo Wang, Siliang Zeng, Alfredo Garcia, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Mingyi Hong

Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization

Read the original on arXiv AI →

The paper introduces SORL, a framework that stabilizes off‑policy reinforcement learning for long‑horizon large language model agents. It identifies two key instability sources—token‑level policy granularity mismatches and high‑variance off‑policy updates—and proposes turn‑level importance sampling and clipping‑triggered normalization to align optimization with multi‑turn interactions. Two instantiations, SO‑PPO and SO‑GRPO, are evaluated on open‑domain, multi‑hop, and medical QA benchmarks, as well as on asynchronous RL for mathematical reasoning, showing improved robustness without the need for early stopping or heuristic tuning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jul 9

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).

By Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin

llms reinforcement-learning multimodal

Hugging Face Trending Papers

Jul 8

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks.

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 9

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

arXiv:2607. 07508v1 Announce Type: cross Abstract: Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs).

By Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jun 2

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).

By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jul 23

REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

arXiv:2607. 19450v1 Announce Type: cross Abstract: Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs).

By Yunjie Chen, Xiaoxin Chen, Fang Wang

llms agents reinforcement-learning efficiency

arXiv AI

Aug 20

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

The paper introduces Reverse‑Turn Policy Optimization (RTPO), a method that restructures multi‑turn agentic reinforcement learning rollouts into sparse reverse trees and updates policies in temporal reverse order. This approach addresses three key instability sources—context mismatch, weak turn‑level credit assignment, and asynchronous policy drift—by aligning each decision with its downstream continuation. Theoretical analysis shows RTPO eliminates context mismatch and drift, reduces credit bias, and converges to recursive optimality, while experiments demonstrate performance gains of 21.50% over trajectory‑level and 10.76% over turn‑level baselines on multi‑turn agentic RL benchmarks.

By Yugu Li, Jimmy Cao, Jianglin Qiao, Siyi Hu

llms agents reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training