arXiv Machine Learning September 18, 2026 By Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim, Joonseok Lee, Jaemoo Choi

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

Read the original on arXiv Machine Learning →

The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Sep 2

Group Adaptive Clipping Policy Optimization

Group Adaptive Clipping Policy Optimization (GAPO) is a plug‑in modification to GRPO methods that adapts the importance‑sampling clipping boundary based on rollout advantage. By allowing rollouts with larger learning signals to receive proportionally greater update headroom, GAPO addresses the limitation of fixed clipping that suppresses rare but informative rollouts. Experiments on Qwen and Llama models show that GAPO consistently improves Pass@1 and Pass@k on math reasoning and coding benchmarks where base model pass rates are low.

By Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft

llms reinforcement-learning benchmarks

arXiv AI

Jun 15

Rethinking the Trust Region in LLM Reinforcement Learning

arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.

By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee

llms reinforcement-learning fine-tuning efficiency

arXiv AI

Jun 3

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

arXiv:2606. 03382v1 Announce Type: cross Abstract: While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments.

By Bingxu Liu, Jiashun Liu, Johan Obando-Ceron, Hao Wang, Runze Liu, Pablo Samuel Castro, Aaron Courville, Ling Pan

llms reinforcement-learning robotics

Hugging Face Trending Papers

Jun 2

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping.

llms reinforcement-learning robotics

arXiv AI

Jun 10

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

arXiv:2606. 10968v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning.

By Renjie Mao, Xiangxin Zhou, Lvfang Tao, Yixin Ding, Yu Shi, Yongguang Lin, Yuheng Wu, Honglin Zhu, Qian Qiu, Wenxi Zhu

llms reinforcement-learning

arXiv Machine Learning

Sep 1

Constrained Group Relative Policy Optimization

arXiv:2602.05863v3 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) remains the dominant critic-free approach for fine-tuning LLMs and VLMs, but its compatibility with const...

By Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azal\'ee Robitaille, Christopher Pal, Liam Paull

llms agents robotics fine-tuning benchmarks safety

One email a morning, machine-written

Related stories

Group Adaptive Clipping Policy Optimization

Rethinking the Trust Region in LLM Reinforcement Learning

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

Constrained Group Relative Policy Optimization