Equivalence between policy gradients and soft Q-learning

Regret of exploratory policy improvement and $q$-learning

arXiv:2411. 01302v2 Announce Type: replace Abstract: We study the convergence of $q$-learning and related algorithms introduced by Jia and Zhou (J.

Soft $Q(\lambda)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces

arXiv:2604. 13780v2 Announce Type: replace-cross Abstract: Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy.

Deep Q-Learning on H\"older Spaces

arXiv:2606. 16846v1 Announce Type: cross Abstract: We study the operator-theoretic core of Q-learning in continuous-time stochastic control with continuous states and actions.

Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

arXiv:2601. 22970v2 Announce Type: replace-cross Abstract: Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment.

Pareto Q-Learning with Reward Machines

arXiv:2606. 19134v1 Announce Type: cross Abstract: We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs).

An Introduction to Q-Learning Part 1

Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approach

arXiv:2511. 03618v2 Announce Type: replace Abstract: In this paper, we formalize the almost sure convergence of $Q$-learning and linear temporal difference (TD) learning with Markovian samples using the Lean 4 theorem prover based on the Mathlib library.

Spectral Analysis of Dueling Q-Learning

arXiv:2607. 08340v1 Announce Type: cross Abstract: Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown.

Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics

arXiv:2606. 02645v1 Announce Type: cross Abstract: Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical explanation is still incomplete.

Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions

The paper introduces state abstractions that preserve the difference of Q‑functions for offline reinforcement learning, aiming to exclude irrelevant dynamics from rich state data. It proposes a dynamic generalization of the R‑learner that uses orthogonal estimation and sparse learning to estimate the Q‑function contrast, achieving faster convergence and consistency under a margin condition. Experiments on simulated and simulator‑augmented real data show variance reductions and demonstrate that the necessary information for sequential decision‑making can be smaller than that required for full state prediction.

An Introduction to Q-Learning Part 2/2

Heavy-Ball Q-Learning with Residual Weighting Correction

arXiv:2606. 27112v1 Announce Type: cross Abstract: This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence.