The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.
By Qipei Chen, Wenye Li, Yule Sun, Ke Wei
arXiv:2611.05562v1 Announce Type: cross
Abstract: In multi-objective reinforcement learning, preferences may be specified only after data collection, requiring one dataset to support a continuum of r...
By Yue Wang
arXiv:2610.32125v1 Announce Type: new
Abstract: Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function \(Q^*\) from a fixed dataset without further in...
By Hao Cheng, Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Chen Zhong
arXiv:2609.36486v1 Announce Type: new
Abstract: We study an unknown-transition finite-horizon Markov decision process (MDP) with a finite collection of known reward functions $\{r^1, r^2, \ldots, r^M...
By Zijun Chen, Zihan Zhang
arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.
By Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang
arXiv:2610.34045v1 Announce Type: cross
Abstract: We study the minimax regret rate for episodic kernel-based reinforcement learning, where the expected reward and transition density belong to a Sobol...
By Haichen Hu, David Simchi-Levi
arXiv:2609.24489v1 Announce Type: new
Abstract: Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets stabilized by target n...
By Hyukjun Yang, Jongchan Park, Narim Jeong, Donghwan Lee
arXiv:2610. 13901v1 Announce Type: cross Abstract: We study the last iterate of standard asynchronous Q-learning from a single trajectory of a finite Markov decision process.
By Yang Peng
Limiting‑Kernel Q(λ) (LKQL) is an off‑policy value estimator that blends n‑step truncation with a long‑horizon approximation based on the limiting kernel. It maintains the computational efficiency of n‑step methods while improving policy evaluation accuracy, especially for long‑horizon tasks. The authors prove faster convergence of LKQL’s operator under aperiodicity and near‑on‑policy conditions, and demonstrate empirical gains on MuJoCo continuous‑control benchmarks.
By Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
arXiv:2609.36945v1 Announce Type: new
Abstract: We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version...
By Zhiwei Wang, Yanxi Chen, Yaliang Li, Bolin Ding
arXiv:2605. 05481v2 Announce Type: replace Abstract: We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy.
By Dillon Sandhu, Ronald Parr
The paper introduces a clipped barrier policy gradient method that incorporates a convex regularizer to manage failure risk in budget-constrained operations. By adding a barrier that grows as the budget depletes and clipping rewards below a threshold, the approach ensures that the learned policy maximizes utility while keeping risk within a bounded margin. The authors prove exponential convergence of the training dynamics under convex regularization and demonstrate the method on several benchmark tasks.
By Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen