We’re releasing a new class of reinforcement learning algorithms, Proximal Policy Optimization (PPO), which perform comparably or better than state-of-the-art approaches while being much simpler to implement and tune. PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance.
arXiv:2406.03894v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due...
arXiv:2606. 23932v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning.
arXiv:2512. 06547v4 Announce Type: replace-cross Abstract: Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting.
arXiv:2511. 02577v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is widely regarded as one of the most successful deep reinforcement learning algorithms, known for its robustness and effectiveness across a range of problems.
arXiv:2406.03678v2 Announce Type: replace Abstract: On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensi...
arXiv:2404. 13879v5 Announce Type: replace Abstract: Uncertainties in transition dynamics pose a critical challenge in reinforcement learning (RL), often resulting in performance degradation of trained policies when deployed on hardware.
arXiv:2512.16565v2 Announce Type: replace-cross Abstract: Reinforcement learning has gained attention for modern Large Language Model post-training. The actor-only variants of Proximal Policy Optimiz...
arXiv:2606. 16733v1 Announce Type: new Abstract: Policy gradient algorithms for language models optimize the same objective $J(\theta) = \mathbb{E}*{\tau \sim p*\theta(\tau)}[R(\tau)]$, which has exactly two factors: the trajectory probability $p_\theta(\tau)$ and the reward $R(\tau)$.
BCPPO is a new variant of Proximal Policy Optimization that uses Bachelier-inspired cost‑prediction networks to generate a smooth penalty based on disagreement among critics. The method keeps temporal‑difference learning unchanged, applies a saturation‑aware controller to manage cost penalties, and deploys only the policy network. Across extensive experiments, BCPPO outperforms comparators in achieving higher mean returns while maintaining lower or comparable CVaR in all tested tasks.
arXiv:2601. 18840v4 Announce Type: replace Abstract: Markov decision problems are most commonly solved via dynamic programming.
The paper identifies a failure mode in Proximal Policy Optimization (PPO) critics called Value Flattening, where state values vary sharply across intermediate states while critic predictions remain flat. The authors analyze this phenomenon theoretically and empirically, linking it to an implicit variance penalty and redundant updates from temporally correlated states. They propose Sparse Proximal Policy Optimization (SP³O), which applies the value loss to only a few well‑separated states, and demonstrate that this sparse supervision mitigates Value Flattening and improves policy performance on Qwen3‑Base across model sizes and evaluation suites.