The paper introduces occupancy-weighted Fitted Q-Evaluation (FQE), a regression-based off‑policy evaluation method that replaces the standard offline distribution weights with a target‑policy discounted occupancy ratio. This weighting aligns the projection norm with the target policy’s dynamics, restoring contraction of the Bellman operator and eliminating the need for Bellman completeness. The authors provide finite‑sample guarantees that separate iteration, statistical, approximation, and ratio‑estimation errors, and show that exact occupancy weighting combined with fitted occupancy‑ratio evaluation yields consistent estimation under coverage without requiring critic‑side completeness.
By Lars van der Laan, Nathan Kallus
arXiv:2607. 05375v1 Announce Type: cross Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation.
By Lars van der Laan, Nathan Kallus
Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class.
arXiv:2609.24489v1 Announce Type: new
Abstract: Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets stabilized by target n...
By Hyukjun Yang, Jongchan Park, Narim Jeong, Donghwan Lee
arXiv:2608.24858v1 Announce Type: new
Abstract: Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characteri...
By Lars van der Laan, Nathan Kallus
Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing mini...
arXiv:2610.14988v1 Announce Type: cross
Abstract: The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlyi...
By Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
arXiv:2609.38955v1 Announce Type: cross
Abstract: Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a...
By Yang chen, Yitan Zhang, Michael Witbrock, Shuyue Hu
The paper presents a convergence framework for deep $V$‑learning over a finite horizon $H$, deriving explicit bounds on policy loss by decomposing the Bellman update error into six residuals. It shows how $L^s$ concentrability controls expected $L^1$ loss, quantifies the impact of shared sampling across horizon levels, and provides optimal and near‑optimal sample allocations for statistical error rates. The work also establishes sharp action‑gap bounds under a margin condition, transfers optimal‑gap results to frozen‑iterate gaps, and offers consistency guarantees for generative‑reset approximate‑ERM procedures with exact action scores.
By Yury Kolomeytsev
arXiv:2609.36390v1 Announce Type: cross
Abstract: Offline reinforcement learning seeks optimal decision rules from previously collected data. In some applications, a decision can be an entire functio...
By Gefei Lin, Rui Miao, Xiaoke Zhang
arXiv:2605. 26078v3 Announce Type: replace Abstract: Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning (RL) that exploits the optimal-transport geometry of action distributions.
By Zhaoyu Zhu, Rui Gao, Shuang Li
arXiv:2501.06926v5 Announce Type: replace
Abstract: Double reinforcement learning (DRL) provides efficient off-policy inference for policy values in nonparametric Markov decision processes (MDPs), bu...
By Lars van der Laan, David Hubbard, Allen Tran, Nathan Kallus, Aur\'{e}lien Bibaut