← Back to all news
arXiv Machine Learning By Zhicheng Cai, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL

Read the original on arXiv Machine Learning →

The paper introduces Trajectory-Aware Policy Optimization (TAPO), a new framework for reinforcement learning that addresses the state distribution shift caused by long-horizon autoregressive rollouts in large language models. TAPO derives a state-distribution correction from the off-policy policy gradient, integrates it into the RL objective, and uses a geometric-mean estimator for variance reduction along with length-aware dynamic clipping to stabilize updates. Experiments on eight competition-level benchmarks show TAPO outperforms existing LLM RL methods, achieving up to 48% improvement on mathematical reasoning and 12% on algorithmic coding tasks while enhancing training stability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 7

Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization

The paper introduces SORL, a framework that stabilizes off‑policy reinforcement learning for long‑horizon large language model agents. It identifies two key instability sources—token‑level policy granularity mismatches and high‑variance off‑policy updates—and proposes turn‑level importance sampling and clipping‑triggered normalization to align optimization with multi‑turn interactions. Two instantiations, SO‑PPO and SO‑GRPO, are evaluated on open‑domain, multi‑hop, and medical QA benchmarks, as well as on asynchronous RL for mathematical reasoning, showing improved robustness without the need for early stopping or heuristic tuning.

By Chenliang Li, Adel Elmahdy, Alex Boyd, Zhongruo Wang, Siliang Zeng, Alfredo Garcia, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Mingyi Hong
llmsagentsreinforcement-learningbenchmarks
More like this →
arXiv AI
Oct 2

Rethinking Probability-Based Reinforcement Learning From Posterior Concentration

The paper introduces the Posterior Concentration Phenomenon (PCP), a length‑dependent failure mode where probability‑based rewards collapse to a narrow interval for long reasoning traces, destabilizing verifier‑free reinforcement learning. To address this, the authors propose RLCPR, a framework that uses uncertainty‑aware data sampling and concentration‑aware regularization to mitigate PCP, improving token efficiency and outperforming state‑of‑the‑art baselines on multiple reasoning benchmarks.

By Shiu-Hong Kao, Yubo Zhao, Zhenyu Tian, Pengzhan Sun, Yicong Li, Angela Yao
llmsreinforcement-learningbenchmarks
More like this →