Hugging Face Trending Papers

Sharp Sample Complexity of KL-Regularized Reinforcement Learning

KL-regularized reinforcement learning, the objective behind maximum-entropy RL and RL from human feedback, is variational inference in the graphical model of "control as inference": the regularized optimal policy is a posterior over trajectories, and the suboptimality of a policy is $β$ times a KL divergence between path measures. However, it is not known how many samples it takes to learn this posterior, even with a generative model.

arXiv AI
6d ago

Sharp Sample Complexity of KL-Regularized Reinforcement Learning

The paper studies the sample complexity of KL‑regularized reinforcement learning, which is equivalent to variational inference in the control‑as‑inference framework. It proves that a plug‑in estimator achieves ε‑optimality with ≥O(KH^3/(ε max{ε,β}) + KH) samples, and establishes a matching lower bound for ε ≤ min{H, H^2/β}. The authors also identify the exact asymptotic constant of the estimator and provide optimal sample allocations for adaptive designs, with simulations confirming the theoretical rates.

By Andrew Liu
arXiv Machine Learning
1d ago

Offline KL-Regularized Reinforcement Learning with Stochastic Transitions

The paper investigates offline KL‑regularized reinforcement learning in finite discounted Markov decision processes with unknown stochastic transitions. It establishes lower bounds on sample complexity for estimators covering the regularized optimum and proposes a pessimistic soft value iteration algorithm that achieves near‑optimal rates under certain temperature and concentrability conditions. The study also highlights the limitations of plug‑in estimators on reference‑policy data and the impact of temperature on sample requirements.

By Andrew Liu
arXiv AI
Oct 27

Provably Efficient Q-learning for Robust Reinforcement Learning

The paper introduces a sample‑efficient asynchronous Q‑learning algorithm for robust reinforcement learning in infinite‑horizon, discounted robust MDPs with total‑variation uncertainty. It proves that, given a single Markovian trajectory, the algorithm achieves an entrywise ε‑approximation of the optimal Q‑function with a sample complexity bounded by a nearly minimax‑optimal expression involving the discount factor, uncertainty level, mixing time, and occupancy probability. This result improves significantly over prior bounds for robust RL.

By Yuchen Jiao, Gen Li
arXiv AI
Nov 6

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability

The paper presents the first ≥O(ε^{-1}) upper bounds for offline contextual bandits with forward-KL regularization under single-policy concentrability, covering both tabular and general function approximation settings. It introduces a convex-analytical approach that leverages pessimism in a novel way, avoiding previous mean value theorem-based proofs. The authors also provide matching lower bounds, showing the tightness of their rates and that the forward-KL sample complexity reverts to the unregularized slow rate when regularization is weak.

By Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Quanquan Gu
arXiv Machine Learning
Nov 2

Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective

The monograph "Statistical Foundations of Reinforcement Learning: A Non-Asymptotic Perspective" surveys recent advances in finite-sample statistical theory for reinforcement learning. It focuses on tabular Markov decision processes and examines three sampling settings—generative model, online, and offline RL—alongside model-based and model-free algorithmic paradigms. The work emphasizes minimax-optimal sample complexity, provides self-contained analyses, and briefly explores extensions such as robust, multi-agent, federated, and human-feedback RL.

By Yuxin Chen, Yuejie Chi, Jianqing Fan, Gen Li, Yuting Wei, Yuling Yan