KL-regularized reinforcement learning, the objective behind maximum-entropy RL and RL from human feedback, is variational inference in the graphical model of "control as inference": the regularized optimal policy is a posterior over trajectories, and the suboptimality of a policy is $β$ times a KL divergence between path measures. However, it is not known how many samples it takes to learn this posterior, even with a generative model.
The paper investigates offline KL‑regularized reinforcement learning in finite discounted Markov decision processes with unknown stochastic transitions. It establishes lower bounds on sample complexity for estimators covering the regularized optimum and proposes a pessimistic soft value iteration algorithm that achieves near‑optimal rates under certain temperature and concentrability conditions. The study also highlights the limitations of plug‑in estimators on reference‑policy data and the impact of temperature on sample requirements.
By Andrew Liu
arXiv:2302.07477v4 Announce Type: replace
Abstract: We study the optimal sample complexity of tabular reinforcement learning for infinite-horizon discounted Markov decision processes. The unrestricte...
By Shengbo Wang, Jose Blanchet, Peter Glynn
arXiv:2610. 13901v1 Announce Type: cross Abstract: We study the last iterate of standard asynchronous Q-learning from a single trajectory of a finite Markov decision process.
By Yang Peng
The paper introduces a sample‑efficient asynchronous Q‑learning algorithm for robust reinforcement learning in infinite‑horizon, discounted robust MDPs with total‑variation uncertainty. It proves that, given a single Markovian trajectory, the algorithm achieves an entrywise ε‑approximation of the optimal Q‑function with a sample complexity bounded by a nearly minimax‑optimal expression involving the discount factor, uncertainty level, mixing time, and occupancy probability. This result improves significantly over prior bounds for robust RL.
By Yuchen Jiao, Gen Li
arXiv:2611. 05562v1 Announce Type: cross Abstract: In multi-objective reinforcement learning, preferences may be specified only after data collection, requiring one dataset to support a continuum of reward trade-offs.
By Yue Wang
arXiv:2606. 25170v1 Announce Type: cross Abstract: We study PAC learning in tabular discounted Markov decision processes with exogenous i.
By Corentin Pla, Hugo Richard, Marc Abeille, Vianney Perchet
arXiv:2609.36486v1 Announce Type: new
Abstract: We study an unknown-transition finite-horizon Markov decision process (MDP) with a finite collection of known reward functions $\{r^1, r^2, \ldots, r^M...
By Zijun Chen, Zihan Zhang
The paper presents the first ≥O(ε^{-1}) upper bounds for offline contextual bandits with forward-KL regularization under single-policy concentrability, covering both tabular and general function approximation settings. It introduces a convex-analytical approach that leverages pessimism in a novel way, avoiding previous mean value theorem-based proofs. The authors also provide matching lower bounds, showing the tightness of their rates and that the forward-KL sample complexity reverts to the unregularized slow rate when regularization is weak.
By Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Quanquan Gu
The paper investigates best‑policy identification in finite‑horizon, risk‑sensitive reinforcement learning using the entropic risk measure. It identifies a gap between known lower bounds ≥ η(e^{|eta|H}) and upper bounds ≤ O(e^{2|eta|H}) for sample complexity, attributing the excess factor to loose concentration bounds for exponential utilities. By employing a forward‑model algorithm with KL‑based exploration bonuses and a novel stopping rule, the authors achieve a sample complexity that matches the lower bound, closing the previously open exponential gap.
By Amer Essakine, Claire Vernade
arXiv:2610.18839v1 Announce Type: cross
Abstract: This paper investigates offline Q-learning for finite-horizon Markov decision processes with Kullback-Leibler (KL) regularization. Recent studies hav...
By Yize Hao, Guangxin Jiang, Shuang Qiu
arXiv:2610.08963v1 Announce Type: new
Abstract: Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come fr...
By Yifan Zhang