The paper investigates whether the policy gradient update used in many reinforcement learning algorithms—specifically the episode-summed, on‑policy update that omits the outer discount factor—can be interpreted as the gradient of any objective function. The authors prove that this update does not necessarily correspond to the gradient of any function and provide a counterexample where the update drives the policy toward a globally pessimal deterministic limit while the parameters diverge. They also highlight that this theoretical ambiguity has led to widespread misstatements in the literature, even in highly cited works from top conferences.
arXiv:2503.22244v3 Announce Type: replace Abstract: Policy gradient methods are among the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical s...
arXiv:2605. 14982v2 Announce Type: replace-cross Abstract: We address the discounted reward setting in reinforcement learning (RL).
The paper introduces Loopless Variance-Reduced Policy Gradient (LVR-PG), a framework that unifies first‑order estimators combining recursive updates with defensive importance sampling, such as PAGE-PG and STORM-PG. It proves that LVR-PG achieves a ∫O(ε⁻²) last‑iterate global convergence rate under weak gradient domination for general parametric policies, using only first‑order updates and minimal assumptions. By extending lower bounds to continuous parametric policies, the authors show an Ω(ε⁻²) sample complexity limit, establishing that LVR-PG is minimax‑optimal up to logarithmic factors and horizon terms.
The paper presents a policy gradient theorem tailored to Cumulative Prospect Theory (CPT) objectives in finite-horizon reinforcement learning, extending the classic policy gradient framework to include distortion-based risk measures. Leveraging this theorem, the authors develop a first‑order policy gradient algorithm that uses a Monte Carlo estimator based on order statistics, providing statistical guarantees and proving asymptotic convergence to first‑order stationary points of the generally nonconvex CPT objective. The work also offers a non‑asymptotic sample complexity bound for reaching an approximate stationary policy and demonstrates the qualitative effects of CPT through simulations, comparing the new first‑order method to existing zeroth‑order approaches.
arXiv:2506. 13862v2 Announce Type: replace-cross Abstract: In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice.
The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.
The paper analyzes the oblivious common-reward multi-player multi-armed bandit (OC-MPMAB), where players act independently without observing others and only share a common reward. It shows that the players’ gradient-bandit updates form an unbiased estimator of a hidden product-policy objective, guaranteeing universal finite‑time first‑order stationarity. The work also identifies conditions under which global optimality cannot be achieved, introduces certificates to recover global optimality under structural assumptions, and establishes corresponding product‑policy gap and pseudo‑regret bounds.
arXiv:2601. 18175v2 Announce Type: replace Abstract: A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories.
arXiv:2602. 01460v3 Announce Type: replace-cross Abstract: Policy-gradient methods are widely used in reinforcement learning, yet training often becomes unstable or slows down as learning progresses.
arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.
The paper addresses policy optimization in online episodic tabular MDPs with adversarial losses and bandit feedback. By employing regularized Q‑functions, the authors eliminate the extra horizon factor present in previous policy‑optimization regret bounds, achieving high‑probability regret of ≥O(√{HS(H+A)T}) for known transitions and ≥O(HS√{AT}) for unknown transitions. These results improve horizon dependence and match the best-known bounds, with extensions to adversarial linear‑mixture MDPs.
arXiv:2609.36945v1 Announce Type: new Abstract: We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version...