arXiv AI By Pengxin Wang, Yuanzhe LI, Yuxin Ren, Huanrui Yang, Jingdi Chen

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

Read the original on arXiv AI →

The paper introduces Stable Perturbation-Robust Policy Optimization (SPrPO), a reinforcement learning method that applies adaptive, sensitivity-aware perturbations during training to improve policy robustness against hidden‑state noise, pruning, and quantization. It defines a perturbation‑robust policy and analyzes conditions for stable monotonic improvement under perturbed updates. Experiments on ALFWorld and WebShop demonstrate that SPrPO enhances robustness across various perturbation types while preserving stable policy optimization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Jun 30

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.

By Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng
arXiv AI
3d ago

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

The paper investigates whether the importance ratio is essential for stable reinforcement learning (RL) in large language models (LLMs). It introduces Likelihood‑Gated Policy Optimization (LGPO), which removes the need to compute old‑policy likelihoods by using a likelihood‑based gating mechanism to enforce a soft trust region. Experiments show that eliminating the importance sampling correction does not hurt stability, while removing the trust region can cause collapse; LGPO matches or exceeds GRPO’s performance, reduces training time by about 18%, and remains stable in fully on‑policy settings where GRPO fails.

By Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho