arXiv Machine Learning

Value Mirror Descent for Reinforcement Learning

arXiv:2604. 06039v2 Announce Type: replace-cross Abstract: Value iteration-type methods have been extensively studied for computing a nearly optimal value function in reinforcement learning (RL).

arXiv Machine Learning
Oct 2

Fast Regularized Policy Mirror Descent with One-Step TD Updates

The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.

By Qipei Chen, Wenye Li, Yule Sun, Ke Wei
arXiv Machine Learning
Sep 30

Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learning

The paper introduces Dually Regularized AIL, a model‑free algorithm for adversarial imitation learning that jointly applies KL policy regularization and a quadratic reward penalty based on expert and learner occupancies. It proves fast convergence rates, achieving a ×O(1/K+1/N) bound on the regularized imitation gap in finite‑horizon MDPs with general function approximation, and establishes the first algorithm to attain ×O(1/ε) sample complexity in both expert demonstrations and online interactions for this regularized objective.

By Hanbin Zhou, Shangzhe Li, Alexander Braverman, Weitong Zhang
arXiv Machine Learning
Oct 23

Primal-Dual Representation Learning for Low-Rank Constrained MDPs

The paper introduces REP-PD-TEG, a representation‑learning algorithm for low‑rank continuous‑state constrained Markov Decision Processes (CMDPs) that achieves optimal ≥O(√K) regret in the soft cumulative‑violation setting while maintaining a ≥O(K^{3/4}) sample bound when exploratory actions are penalised. For the stricter hard cumulative‑violation scenario, REP-PD-H-TEG is proposed, which uses a bonus‑corrected lower‑confidence utility certificate and a stabilized dual search to guarantee both regret and hard‑violation bounds. Together, these methods provide the first theoretically guaranteed representation‑learning framework for low‑rank continuous‑state CMDPs with both soft and hard violation controls.

By Chenhao Zhou, Chao Zhang, Wensong Bai, Hanbin Zhao, Hui Qian
arXiv AI
Nov 6

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability

The paper presents the first ≥O(ε^{-1}) upper bounds for offline contextual bandits with forward-KL regularization under single-policy concentrability, covering both tabular and general function approximation settings. It introduces a convex-analytical approach that leverages pessimism in a novel way, avoiding previous mean value theorem-based proofs. The authors also provide matching lower bounds, showing the tightness of their rates and that the forward-KL sample complexity reverts to the unregularized slow rate when regularization is weak.

By Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Quanquan Gu