The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.
By Qipei Chen, Wenye Li, Yule Sun, Ke Wei
arXiv:2608. 19587v1 Announce Type: new Abstract: While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored.
By Zhiqiang Tan
arXiv:2609.39595v1 Announce Type: new
Abstract: Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Ne...
By Hanyng Peng, Hui Wang, Yue Yu
While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored. Existing analyses often rely on double-loop architectures and invoke a linear entropy penalty.
arXiv:2606. 05967v1 Announce Type: cross Abstract: In this paper, we study the finite-time behavior of the TD(0) temporal-difference method with linear function approximation (LFA).
By Ziad Kobeissi (L2S), \'Elo\"ise Berthier (U2IS)
arXiv:2606. 08783v1 Announce Type: cross Abstract: Orthogonalized momentum updates, as used in Muon-style optimizers, have recently shown strong empirical stability in large-scale deep learning.
By Ganzhao Yuan
In this paper, we study the finite-time behavior of the TD(0) temporal-difference method with linear function approximation (LFA). We consider on-policy independent and identically distributed (i.
The paper investigates how distribution mismatch—when the sampling distribution used to estimate policy gradient updates differs from the discounted state distribution required by theory—affects on‑policy policy gradient methods. For tabular direct and softmax parameterizations, it proves global convergence of objective values under biased updates and, for tabular softmax policies, derives exact leading‑order coefficients of asymptotic objective errors, providing a criterion to determine whether mismatch speeds up or slows convergence. The analysis is extended to general policy parameterizations, yielding explicit bounds on state distribution mismatch and normalized policy‑gradient mismatch in both episodic and continuing MDPs, showing these mismatches vanish at least linearly as the discount factor approaches one and bounding the squared norm of the normalized exact policy‑gradient direction under smoothness assumptions.
By Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan
arXiv:2506. 13862v2 Announce Type: replace-cross Abstract: In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice.
By Alex Davey, Alena Shilova, Brahim Driss, Riad Akrour
arXiv:2609.07666v1 Announce Type: new
Abstract: Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order...
By Yuyang Wang, Haoyu Yao, Pengcheng Xie
arXiv:2607. 28390v1 Announce Type: new Abstract: Constrained Markov Decision Processes (CMDPs) provide a natural framework for reinforcement learning in safety-critical applications, where agents maximize long-term reward while satisfying long-term constraints.
By Ankur Naskar, Vaneet Aggarwal
arXiv:2610. 14567v2 Announce Type: replace-cross Abstract: This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD).
By Joseph Keslin, Wei Biao Wu