arXiv Machine Learning By Andrew Liu

Offline KL-Regularized Reinforcement Learning with Stochastic Transitions

Read the original on arXiv Machine Learning →

The paper investigates offline KL‑regularized reinforcement learning in finite discounted Markov decision processes with unknown stochastic transitions. It establishes lower bounds on sample complexity for estimators covering the regularized optimum and proposes a pessimistic soft value iteration algorithm that achieves near‑optimal rates under certain temperature and concentrability conditions. The study also highlights the limitations of plug‑in estimators on reference‑policy data and the impact of temperature on sample requirements.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
6d ago

Sharp Sample Complexity of KL-Regularized Reinforcement Learning

The paper studies the sample complexity of KL‑regularized reinforcement learning, which is equivalent to variational inference in the control‑as‑inference framework. It proves that a plug‑in estimator achieves ε‑optimality with ≥O(KH^3/(ε max{ε,β}) + KH) samples, and establishes a matching lower bound for ε ≤ min{H, H^2/β}. The authors also identify the exact asymptotic constant of the estimator and provide optimal sample allocations for adaptive designs, with simulations confirming the theoretical rates.

By Andrew Liu
Hugging Face Trending Papers
Nov 10

Sharp Sample Complexity of KL-Regularized Reinforcement Learning

KL-regularized reinforcement learning, the objective behind maximum-entropy RL and RL from human feedback, is variational inference in the graphical model of "control as inference": the regularized optimal policy is a posterior over trajectories, and the suboptimality of a policy is $β$ times a KL divergence between path measures. However, it is not known how many samples it takes to learn this posterior, even with a generative model.

arXiv AI
Nov 6

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability

The paper presents the first ≥O(ε^{-1}) upper bounds for offline contextual bandits with forward-KL regularization under single-policy concentrability, covering both tabular and general function approximation settings. It introduces a convex-analytical approach that leverages pessimism in a novel way, avoiding previous mean value theorem-based proofs. The authors also provide matching lower bounds, showing the tightness of their rates and that the forward-KL sample complexity reverts to the unregularized slow rate when regularization is weak.

By Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Quanquan Gu