arXiv Machine Learning

Statistical analysis of Inverse Entropy-regularized Reinforcement Learning

The paper introduces a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves the non-uniqueness of reward functions by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. It models expert demonstrations as a Markov chain, estimates the expert policy via penalized maximum likelihood, and provides high-probability bounds on the excess Kullback–Leibler divergence between the estimated and true policies. These results yield non-asymptotic minimax optimal convergence rates for the least-squares reward function, highlighting the trade-offs among smoothing, model complexity, and sample size.

arXiv Machine Learning
Sep 30

Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learning

The paper introduces Dually Regularized AIL, a model‑free algorithm for adversarial imitation learning that jointly applies KL policy regularization and a quadratic reward penalty based on expert and learner occupancies. It proves fast convergence rates, achieving a ×O(1/K+1/N) bound on the regularized imitation gap in finite‑horizon MDPs with general function approximation, and establishes the first algorithm to attain ×O(1/ε) sample complexity in both expert demonstrations and online interactions for this regularized objective.

By Hanbin Zhou, Shangzhe Li, Alexander Braverman, Weitong Zhang
arXiv Machine Learning
Oct 23

To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning

The paper examines the trade-off between using privileged expert distillation and standard reinforcement learning in partially observable settings. By analyzing a theoretical model called the perturbed Block MDP and running experiments on simulated locomotion tasks, it finds that the effectiveness of distillation depends on the stochasticity of latent dynamics and that the optimal latent policy is not always the best to distill. These insights provide guidelines for better leveraging privileged information to improve policy learning efficiency.

By Yuda Song, Dhruv Rohatgi, Aarti Singh, J. Andrew Bagnell