The paper introduces occupancy-weighted Fitted Q-Evaluation (FQE), a regression-based off‑policy evaluation method that replaces the standard offline distribution weights with a target‑policy discounted occupancy ratio. This weighting aligns the projection norm with the target policy’s dynamics, restoring contraction of the Bellman operator and eliminating the need for Bellman completeness. The authors provide finite‑sample guarantees that separate iteration, statistical, approximation, and ratio‑estimation errors, and show that exact occupancy weighting combined with fitted occupancy‑ratio evaluation yields consistent estimation under coverage without requiring critic‑side completeness.
By Lars van der Laan, Nathan Kallus
arXiv:2607. 05375v1 Announce Type: cross Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation.
By Lars van der Laan, Nathan Kallus
Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class.
arXiv:2608.24858v1 Announce Type: new
Abstract: Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characteri...
By Lars van der Laan, Nathan Kallus
Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing mini...
arXiv:2605. 26078v3 Announce Type: replace Abstract: Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning (RL) that exploits the optimal-transport geometry of action distributions.
By Zhaoyu Zhu, Rui Gao, Shuang Li
arXiv:2501.06926v5 Announce Type: replace
Abstract: Double reinforcement learning (DRL) provides efficient off-policy inference for policy values in nonparametric Markov decision processes (MDPs), bu...
By Lars van der Laan, David Hubbard, Allen Tran, Nathan Kallus, Aur\'{e}lien Bibaut
arXiv:2603. 09344v3 Announce Type: replace Abstract: Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift.
By Hongqiang Lin, Zhenghui Fu, Weihao Tang, Pengfei Wang, Yiding Sun, Qixian Huang, Dongxu Zhang
arXiv:2608. 03108v1 Announce Type: new Abstract: Offline reinforcement learning (offline RL) can benefit from nearby out-of-distribution (OOD) actions, but estimation errors at these actions may be amplified by bootstrapping.
By Yi Yang, Zhennan Chen, Mingfeng Lv, Hanlei Li, Zhengsen Ruan, Lvqing Yang
arXiv:2608.22636v1 Announce Type: cross
Abstract: Q-learning with linear function approximation can be unstable because an arbitrary approximation architecture need not preserve the Bellman contracti...
By Shengbo Wang
arXiv:2608. 14401v1 Announce Type: cross Abstract: In offline RL, estimating the optimal action-value function $Q^*$ can be formulated as solving the optimal Bellman equation based solely on offline observations.
By Xiaohong Chen, Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Chen Zhong
arXiv:2506. 13862v2 Announce Type: replace-cross Abstract: In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice.
By Alex Davey, Alena Shilova, Brahim Driss, Riad Akrour