On the Convergence of Semi-Gradient TD: A Mean Field Perspective

Equivalence between policy gradients and soft Q-learning

Mean Field Reinforcement Learning

arXiv:2607. 01525v1 Announce Type: cross Abstract: This monograph provides an introduction to mean field reinforcement learning through the lens of Markov decision processes arising from large-population stochastic control with mean field interactions and common noise.

Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation

The paper investigates model‑free robust Q‑learning with χ² uncertainty sets and linear function approximation, using data from a single trajectory of an unknown nominal MDP. It introduces a variational reformulation of the robust Bellman target and a blockwise frozen‑target scheme to overcome estimation and non‑contractivity challenges, and proves a finite‑time error bound for every discount factor γ in (0,1). A neural‑network experiment demonstrates the practical use of the variational target in a continuous‑state nonlinear‑control task.

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

The paper investigates continuous‑time stochastic control problems with unknown drift and running reward functions, using an exploratory reinforcement learning framework that incorporates relaxed controls and entropy regularization. It develops policy‑iteration algorithms based on probabilistic representations of the optimal value function and its gradient, proving convergence and demonstrating performance through numerical examples. The study also extends to a special case with control‑dependent diffusion, requiring a Hessian representation.

Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics

arXiv:2606. 02645v1 Announce Type: cross Abstract: Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical explanation is still incomplete.

Regret of exploratory policy improvement and $q$-learning

arXiv:2411. 01302v2 Announce Type: replace Abstract: We study the convergence of $q$-learning and related algorithms introduced by Jia and Zhou (J.

Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis

arXiv:2503. 18607v2 Announce Type: replace-cross Abstract: We introduce the Switching Non-Stationary Markov Decision Process (SNS-MDP) framework, in which the environment transitions among a finite set of MDPs governed by a latent Markov chain while the agent observes only the external state.

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

The paper introduces a continuous‑time policy gradient framework that works with discrete observation data. It derives performance difference lemmas and policy gradient formulas for randomized and parametrized policies, proving exponential convergence for an entropy‑regularized policy flow and providing explicit convergence bounds for discrete updates. By integrating with PhiBE, the authors create model‑free actor‑critic algorithms that maintain the continuous‑time structure, and demonstrate their effectiveness on both linear‑quadratic and nonlinear robotic control tasks.

Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning

arXiv:2605. 16103v2 Announce Type: replace Abstract: Q-learning is known to suffer from overestimation bias: because the Bellman update maximizes noisy or imperfect action-value estimates, positive errors can be selected and propagated, causing learned values to exceed the true optimal values.

Deep Q-Learning on H\"older Spaces

arXiv:2606. 16846v1 Announce Type: cross Abstract: We study the operator-theoretic core of Q-learning in continuous-time stochastic control with continuous states and actions.

Geometrically Averaged Hard Target Updates for Linear Q-Learning

arXiv:2606. 10835v1 Announce Type: cross Abstract: Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning.

Spectral Analysis of Dueling Q-Learning

arXiv:2607. 08340v1 Announce Type: cross Abstract: Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown.