← Back to all news
arXiv Machine Learning By Chris Nota, Philip S. Thomas

Is the Policy Gradient a Gradient?

Read the original on arXiv Machine Learning →

The paper investigates whether the policy gradient update used in many reinforcement learning algorithms—specifically the episode-summed, on‑policy update that omits the outer discount factor—can be interpreted as the gradient of any objective function. The authors prove that this update does not necessarily correspond to the gradient of any function and provide a counterexample where the update drives the policy toward a globally pessimal deterministic limit while the parameters diverge. They also highlight that this theoretical ambiguity has led to widespread misstatements in the literature, even in highly cited works from top conferences.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 12

Variance-Reduced Policy Gradient is Nearly Minimax-Optimal

The paper introduces Loopless Variance-Reduced Policy Gradient (LVR-PG), a framework that unifies first‑order estimators combining recursive updates with defensive importance sampling, such as PAGE-PG and STORM-PG. It proves that LVR-PG achieves a ∫O(ε⁻²) last‑iterate global convergence rate under weak gradient domination for general parametric policies, using only first‑order updates and minimal assumptions. By extending lower bounds to continuous parametric policies, the authors show an Ω(ε⁻²) sample complexity limit, establishing that LVR-PG is minimax‑optimal up to logarithmic factors and horizon terms.

By Leonardo Cesani, Matteo Papini, Marcello Restelli
reinforcement-learningbenchmarks
More like this →
arXiv AI
Sep 10

Policy Gradients for Cumulative Prospect Theory in Reinforcement Learning

The paper presents a policy gradient theorem tailored to Cumulative Prospect Theory (CPT) objectives in finite-horizon reinforcement learning, extending the classic policy gradient framework to include distortion-based risk measures. Leveraging this theorem, the authors develop a first‑order policy gradient algorithm that uses a Monte Carlo estimator based on order statistics, providing statistical guarantees and proving asymptotic convergence to first‑order stationary points of the generally nonconvex CPT objective. The work also offers a non‑asymptotic sample complexity bound for reaching an approximate stationary policy and demonstrates the qualitative effects of CPT through simulations, comparing the new first‑order method to existing zeroth‑order approaches.

By Olivier Lepel, Anas Barakat
reinforcement-learning
More like this →
arXiv Machine Learning
Oct 2

Fast Regularized Policy Mirror Descent with One-Step TD Updates

The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.

By Qipei Chen, Wenye Li, Yule Sun, Ke Wei
reinforcement-learning
More like this →