Policy Gradient with PyTorch

Variance reduction for policy gradient with action-dependent factorized baselines

Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

arXiv:2605. 18591v2 Announce Type: replace Abstract: Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix.

Equivalence between policy gradients and soft Q-learning

Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition

arXiv:2605. 14982v2 Announce Type: replace-cross Abstract: We address the discounted reward setting in reinforcement learning (RL).

Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination

arXiv:2605. 04568v3 Announce Type: replace-cross Abstract: State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning.

Proximal Policy Optimization (PPO)

Proximal Policy Optimization for Amortized Discrete Sampling

arXiv:2606. 15793v1 Announce Type: cross Abstract: This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework.

Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertainty

arXiv:2404. 13879v5 Announce Type: replace Abstract: Uncertainties in transition dynamics pose a critical challenge in reinforcement learning (RL), often resulting in performance degradation of trained policies when deployed on hardware.

Evolved Policy Gradients

We’re releasing an experimental metalearning approach called Evolved Policy Gradients, a method that evolves the loss function of learning agents, which can enable fast training on novel tasks. Agents trained with EPG can succeed at basic tasks at test time that were outside their training regime, like learning to navigate to an object on a different side of the room from where it was placed during training.

Proximal Policy Optimization

We’re releasing a new class of reinforcement learning algorithms, Proximal Policy Optimization (PPO), which perform comparably or better than state-of-the-art approaches while being much simpler to implement and tune. PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance.

Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence

arXiv:2601. 18840v4 Announce Type: replace Abstract: Markov decision problems are most commonly solved via dynamic programming.

Self-Distilled Policy Gradient

arXiv:2606. 04036v1 Announce Type: new Abstract: On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning.