The paper studies the sample complexity of KL‑regularized reinforcement learning, which is equivalent to variational inference in the control‑as‑inference framework. It proves that a plug‑in estimator achieves ε‑optimality with ≥O(KH^3/(ε max{ε,β}) + KH) samples, and establishes a matching lower bound for ε ≤ min{H, H^2/β}. The authors also identify the exact asymptotic constant of the estimator and provide optimal sample allocations for adaptive designs, with simulations confirming the theoretical rates.
By Andrew Liu
The paper investigates offline KL‑regularized reinforcement learning in finite discounted Markov decision processes with unknown stochastic transitions. It establishes lower bounds on sample complexity for estimators covering the regularized optimum and proposes a pessimistic soft value iteration algorithm that achieves near‑optimal rates under certain temperature and concentrability conditions. The study also highlights the limitations of plug‑in estimators on reference‑policy data and the impact of temperature on sample requirements.
By Andrew Liu
arXiv:2302.07477v4 Announce Type: replace
Abstract: We study the optimal sample complexity of tabular reinforcement learning for infinite-horizon discounted Markov decision processes. The unrestricte...
By Shengbo Wang, Jose Blanchet, Peter Glynn
arXiv:2610. 13901v1 Announce Type: cross Abstract: We study the last iterate of standard asynchronous Q-learning from a single trajectory of a finite Markov decision process.
By Yang Peng
The paper introduces a sample‑efficient asynchronous Q‑learning algorithm for robust reinforcement learning in infinite‑horizon, discounted robust MDPs with total‑variation uncertainty. It proves that, given a single Markovian trajectory, the algorithm achieves an entrywise ε‑approximation of the optimal Q‑function with a sample complexity bounded by a nearly minimax‑optimal expression involving the discount factor, uncertainty level, mixing time, and occupancy probability. This result improves significantly over prior bounds for robust RL.
By Yuchen Jiao, Gen Li
arXiv:2611. 05562v1 Announce Type: cross Abstract: In multi-objective reinforcement learning, preferences may be specified only after data collection, requiring one dataset to support a continuum of reward trade-offs.
By Yue Wang