arXiv:2406.03894v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due...
The paper investigates how reusing past samples can improve the sample efficiency of Proximal Policy Optimization (PPO). Two variants, wPPO-U and wPPO-BH, are introduced within a multiple importance weighting framework, each reusing data from recent iterations while preserving core PPO mechanics. The authors derive theoretical policy improvement bounds for both variants and empirically evaluate their impact on continuous control tasks.
arXiv:2602. 05379v2 Announce Type: replace-cross Abstract: Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization.
arXiv:2605. 05481v2 Announce Type: replace Abstract: We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy.
We’re releasing a new class of reinforcement learning algorithms, Proximal Policy Optimization (PPO), which perform comparably or better than state-of-the-art approaches while being much simpler to implement and tune. PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance.
The paper extends theoretical guarantees for offline reinforcement learning to parameterized policy classes over large or continuous action spaces, overcoming limitations of prior algorithms that required finite action sets and state‑wise mirror descent. By addressing contextual coupling and linking mirror descent to natural policy gradients, the authors provide new analyses, algorithmic insights, and a surprising connection between offline RL and imitation learning.
arXiv:2605. 23415v2 Announce Type: replace Abstract: Reinforcement learning has long struggled with poor sample efficiency.
The paper introduces PSPO, a method for offline policy optimization that treats the dynamics model as a random variable, enabling controlled exploration of out‑of‑distribution (OOD) regions while maintaining robustness. By alternating updates of the posterior distribution and the policy, PSPO achieves a regularized optimization algorithm with convergence guarantees. Experiments on standard benchmarks show that PSPO outperforms state‑of‑the‑art baselines and retains a pessimism‑free property without sacrificing robustness.
ReflectPO introduces a reinforcement learning framework that enables large language model agents to recover from errors during long-horizon tasks involving external tools. The method uses Reflection-Guided Sampling to generate recovery hints from branch differences and repeated visits, and Tree-Structured Credit Assignment to allocate action-level credit based on parent–child reward differences and recovery gains. Experiments on two tool-use benchmarks demonstrate that ReflectPO outperforms existing baselines and improves agents’ ability to detect and correct deviations, thereby advancing autonomous error recovery in complex real-world environments.
arXiv:2603. 03955v2 Announce Type: replace Abstract: Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation.
arXiv:2512. 06547v4 Announce Type: replace-cross Abstract: Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting.
The paper introduces Trust Region Q Adjoint Matching (TRQAM), an off‑policy fine‑tuning algorithm that stabilizes reinforcement learning of pretrained flow policies by adaptively controlling the path‑space KL divergence between the fine‑tuned and pretrained policies. TRQAM achieves this through projected dual descent, which adjusts a trust‑region parameter λ during sampling and ensures the KL penalty is exactly weighted in the stochastic optimal control objective. Experiments on 50 OGBench tasks show TRQAM consistently outperforms prior methods, achieving a 68% success rate in offline RL versus 46% for the strongest baseline.