arXiv AI October 12, 2026 By Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

Read the original on arXiv AI →

The paper introduces Generalized Proximal Policy Optimization (GPPO), a unified framework that extends PPO by incorporating $f$-divergence trust-region theory and novel estimator topologies for large language model reinforcement learning. GPPO demonstrates that different divergences produce distinct policy update dynamics and that estimator design significantly impacts performance. Experiments on eight math and coding benchmarks show GPPO surpasses standard GRPO by up to 50% in math and 14% in coding, achieving state‑of‑the‑art results.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 14

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

arXiv:2607. 10169v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities.

By Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 15

Rethinking the Trust Region in LLM Reinforcement Learning

arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.

By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee

llms reinforcement-learning fine-tuning efficiency

arXiv Machine Learning

Jun 10

Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

arXiv:2606. 11025v1 Announce Type: new Abstract: Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation.

By Bowen Ping, Xiangxin Zhou, Penghui Qi, Minnan Luo, Liefeng Bo, Tianyu Pang

diffusion reinforcement-learning safety

arXiv AI

Jun 3

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

arXiv:2606. 03382v1 Announce Type: cross Abstract: While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments.

By Bingxu Liu, Jiashun Liu, Johan Obando-Ceron, Hao Wang, Runze Liu, Pablo Samuel Castro, Aaron Courville, Ling Pan

llms reinforcement-learning robotics

arXiv AI

Aug 11

Parameter Exploration for RLVR via Variational Learning

arXiv:2608. 09805v1 Announce Type: cross Abstract: Exploration has been a focus of reinforcement learning research for a long time.

By Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych

llms reinforcement-learning

arXiv AI

Jun 29

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.

By Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang

llms reinforcement-learning

One email a morning, machine-written

Related stories

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

Rethinking the Trust Region in LLM Reinforcement Learning

Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

Parameter Exploration for RLVR via Variational Learning

Trust Region Masking for Long-Horizon LLM Reinforcement Learning