arXiv Machine Learning
Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
5d ago
Is the Policy Gradient a Gradient?
The paper investigates whether the policy gradient update used in many reinforcement learning algorithms—specifically the episode-summed, on‑policy update that omits the outer discount factor—can be interpreted as the gradient of any objective function. The authors prove that this update does not necessarily correspond to the gradient of any function and provide a counterexample where the update drives the policy toward a globally pessimal deterministic limit while the parameters diverge. They also highlight that this theoretical ambiguity has led to widespread misstatements in the literature, even in highly cited works from top conferences.
arXiv Machine Learning
Oct 3
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
arXiv:2505. 03155v2 Announce Type: replace Abstract: Policy gradient (PG) methods have played an essential role in the empirical successes of reinforcement learning.
arXiv AI
Aug 3
StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
arXiv:2506. 13862v2 Announce Type: replace-cross Abstract: In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice.
arXiv Machine Learning
Oct 2
Fast Regularized Policy Mirror Descent with One-Step TD Updates
The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.
arXiv Machine Learning
Jun 19
Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
arXiv:2605. 05481v2 Announce Type: replace Abstract: We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy.
arXiv AI
Oct 12
Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
The paper introduces Loopless Variance-Reduced Policy Gradient (LVR-PG), a framework that unifies first‑order estimators combining recursive updates with defensive importance sampling, such as PAGE-PG and STORM-PG. It proves that LVR-PG achieves a ∫O(ε⁻²) last‑iterate global convergence rate under weak gradient domination for general parametric policies, using only first‑order updates and minimal assumptions. By extending lower bounds to continuous parametric policies, the authors show an Ω(ε⁻²) sample complexity limit, establishing that LVR-PG is minimax‑optimal up to logarithmic factors and horizon terms.