arXiv Machine Learning

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.

arXiv Machine Learning

Sep 2

Group Adaptive Clipping Policy Optimization

Group Adaptive Clipping Policy Optimization (GAPO) is a plug‑in modification to GRPO methods that adapts the importance‑sampling clipping boundary based on rollout advantage. By allowing rollouts with larger learning signals to receive proportionally greater update headroom, GAPO addresses the limitation of fixed clipping that suppresses rare but informative rollouts. Experiments on Qwen and Llama models show that GAPO consistently improves Pass@1 and Pass@k on math reasoning and coding benchmarks where base model pass rates are low.

By Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft

llms reinforcement-learning benchmarks

arXiv AI

Jun 15

Rethinking the Trust Region in LLM Reinforcement Learning

arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.

By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee

llms reinforcement-learning fine-tuning efficiency

arXiv AI

Jun 3

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

arXiv:2606. 03382v1 Announce Type: cross Abstract: While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments.

By Bingxu Liu, Jiashun Liu, Johan Obando-Ceron, Hao Wang, Runze Liu, Pablo Samuel Castro, Aaron Courville, Ling Pan

llms reinforcement-learning robotics

Hugging Face Trending Papers

Jun 2

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping.

llms reinforcement-learning robotics

arXiv AI

Jun 10

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

arXiv:2606. 10968v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning.

By Renjie Mao, Xiangxin Zhou, Lvfang Tao, Yixin Ding, Yu Shi, Yongguang Lin, Yuheng Wu, Honglin Zhu, Qian Qiu, Wenxi Zhu

llms reinforcement-learning

arXiv Machine Learning

Sep 1

Constrained Group Relative Policy Optimization

arXiv:2602.05863v3 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) remains the dominant critic-free approach for fine-tuning LLMs and VLMs, but its compatibility with const...

By Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azal\'ee Robitaille, Christopher Pal, Liam Paull

llms agents robotics fine-tuning benchmarks safety

arXiv Machine Learning

Jun 16

A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

arXiv:2606. 16154v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but GRPO-style optimization remains prone to collapse.

By Prasanth YSS, Zhichen Ren, Rasa Hosseinzadeh, Ilan Gofman, Yuqi Chen, Zhaoyan Liu, Guangwei Yu, Jesse C. Cresswell, Satya Krishna Gorti

reinforcement-learning benchmarks

arXiv AI

Jun 10

SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration

arXiv:2606. 10228v1 Announce Type: cross Abstract: Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains.

By Kaustubh Mani, Yann Pequignot, Vincent Mai, Liam Paull

agents reinforcement-learning safety

arXiv Machine Learning

Jun 25

Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning

arXiv:2606. 25526v1 Announce Type: new Abstract: Cooperative multi-agent reinforcement learning assumes each agent shares the same reward function and can be trained effectively using the Trust Region framework of single-agent.

By Bang Giang Le, Viet Cuong Ta

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

Rethinking the Divergence Regularization in LLM RL

arXiv:2606. 09821v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a key component of post-training large language models (LLMs).

By Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, Tianyu Pang

llms reinforcement-learning

arXiv Computation and Language

Aug 25

Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization

arXiv:2608.23311v1 Announce Type: new Abstract: Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer...

By Xianlei Zhou, Xiangdi Meng, Yu He, Tianyu Qi, Shuyan Guan, Xianli Zhang, Jian Zhang, Xin Li, Qika Lin, Jun Liu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 5

GIPO: Gaussian Importance Sampling Policy Optimization

arXiv:2603. 03955v2 Announce Type: replace Abstract: Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation.

By Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Yanjie Li, Baigui Sun, Yang Liu

agents reinforcement-learning multimodal benchmarks safety

Related stories