arXiv:2608.24858v1 Announce Type: new Abstract: Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characteri...
Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing mini...
arXiv:2607. 05375v1 Announce Type: cross Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation.
The paper introduces occupancy-weighted Fitted Q-Evaluation (FQE), a regression-based off‑policy evaluation method that replaces the standard offline distribution weights with a target‑policy discounted occupancy ratio. This weighting aligns the projection norm with the target policy’s dynamics, restoring contraction of the Bellman operator and eliminating the need for Bellman completeness. The authors provide finite‑sample guarantees that separate iteration, statistical, approximation, and ratio‑estimation errors, and show that exact occupancy weighting combined with fitted occupancy‑ratio evaluation yields consistent estimation under coverage without requiring critic‑side completeness.
Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class.
arXiv:2501.06926v5 Announce Type: replace Abstract: Double reinforcement learning (DRL) provides efficient off-policy inference for policy values in nonparametric Markov decision processes (MDPs), bu...
The paper presents a convergence framework for deep $V$‑learning over a finite horizon $H$, deriving explicit bounds on policy loss by decomposing the Bellman update error into six residuals. It shows how $L^s$ concentrability controls expected $L^1$ loss, quantifies the impact of shared sampling across horizon levels, and provides optimal and near‑optimal sample allocations for statistical error rates. The work also establishes sharp action‑gap bounds under a margin condition, transfers optimal‑gap results to frozen‑iterate gaps, and offers consistency guarantees for generative‑reset approximate‑ERM procedures with exact action scores.
arXiv:2609.24489v1 Announce Type: new Abstract: Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets stabilized by target n...
Limiting‑Kernel Q(λ) (LKQL) is an off‑policy value estimator that blends n‑step truncation with a long‑horizon approximation based on the limiting kernel. It maintains the computational efficiency of n‑step methods while improving policy evaluation accuracy, especially for long‑horizon tasks. The authors prove faster convergence of LKQL’s operator under aperiodicity and near‑on‑policy conditions, and demonstrate empirical gains on MuJoCo continuous‑control benchmarks.
The paper introduces a ground‑truth‑first benchmarking framework for reinforcement learning that uses converse optimality to construct tasks with certified optimal value functions and policies. It unifies two benchmarking modes: sensitivity testing with continuous‑time generators to study robustness to task variation, and oracle‑referenced comparison with discrete‑time generators to evaluate policies via optimality gap, regret, and policy error. The methodology is applied to planar arms, nonholonomic vehicles, and quadrotor tasks, and extends to non‑smooth systems, raw‑action interfaces, and partially observed settings, with code and diagnostics publicly available.
The paper introduces a ground‑truth‑first benchmarking framework for reinforcement learning that uses converse optimality to construct tasks with certified optimal value functions and policies. It unifies sensitivity benchmarking—sampling certified environments from continuous‑time generators—and oracle‑referenced comparison—using native discrete‑time generators to evaluate policies via optimality gap, regret, and policy error. The methodology extends to non‑smooth, non‑affine, and partially observed settings, and the authors provide code and diagnostics on a public project page.
arXiv:2610.02828v1 Announce Type: new Abstract: Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL r...