Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approach

End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions

arXiv:2603. 23461v2 Announce Type: replace Abstract: We study reinforcement learning (RL) with linear function approximation in Markov Decision Processes (MDPs) satisfying \emph{linear Bellman completeness} -- a fundamental setting where the Bellman backup of any linear value function remains linear.

Regret of exploratory policy improvement and $q$-learning

arXiv:2411. 01302v2 Announce Type: replace Abstract: We study the convergence of $q$-learning and related algorithms introduced by Jia and Zhou (J.

Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis

arXiv:2503. 18607v2 Announce Type: replace-cross Abstract: We introduce the Switching Non-Stationary Markov Decision Process (SNS-MDP) framework, in which the environment transitions among a finite set of MDPs governed by a latent Markov chain while the agent observes only the external state.

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^\pi$-Realizability and Concentrability

arXiv:2510. 03494v2 Announce Type: replace Abstract: We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization.

Commit to the Bit: Reactive Reinforcement Learning Done Right

arXiv:2605. 28276v2 Announce Type: replace Abstract: Reinforcement learning algorithms are commonly analyzed (and designed) under the Markov assumption.

Non-Asymptotic Best Policy Identification Guarantees in Online Reinforcement Learning

arXiv:2607. 17201v1 Announce Type: cross Abstract: In this work we study the Best Policy Identification (BPI) problem in online, tabular Reinforcement Learning.

Heavy-Ball Q-Learning with Residual Weighting Correction

arXiv:2606. 27112v1 Announce Type: cross Abstract: This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence.

Spectral Analysis of Dueling Q-Learning

arXiv:2607. 08340v1 Announce Type: cross Abstract: Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown.

StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent

arXiv:2506. 13862v2 Announce Type: replace-cross Abstract: In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice.

Heavy-Ball Q-Learning with Residual Weighting Correction

This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning.

Equivalence between policy gradients and soft Q-learning

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

arXiv:2401.15719v5 Announce Type: replace-cross Abstract: We prove a non-asymptotic central limit theorem for vector-valued martingale differences using Stein's method, and use Poisson's equation to...