The paper introduces a ground‑truth‑first benchmarking framework for reinforcement learning that uses converse optimality to construct tasks with certified optimal value functions and policies. It unifies two benchmarking modes: sensitivity testing with continuous‑time generators to study robustness to task variation, and oracle‑referenced comparison with discrete‑time generators to evaluate policies via optimality gap, regret, and policy error. The methodology is applied to planar arms, nonholonomic vehicles, and quadrotor tasks, and extends to non‑smooth systems, raw‑action interfaces, and partially observed settings, with code and diagnostics publicly available.
Limiting‑Kernel Q(λ) (LKQL) is an off‑policy value estimator that blends n‑step truncation with a long‑horizon approximation based on the limiting kernel. It maintains the computational efficiency of n‑step methods while improving policy evaluation accuracy, especially for long‑horizon tasks. The authors prove faster convergence of LKQL’s operator under aperiodicity and near‑on‑policy conditions, and demonstrate empirical gains on MuJoCo continuous‑control benchmarks.
By Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
The paper introduces Policy Gradient Penalty (PGP), a single‑loop policy‑space method that enforces convex occupancy‑measure constraints via quadratic‑penalty regularization. PGP constructs pseudo‑rewards to estimate gradients of the penalized objective and uses the classical Policy Gradient Theorem, establishing smoothness and global last‑iterate convergence guarantees for an ε‑optimal constrained entropy value with ε‑bounded constraint violation. The authors validate PGP with ablations on a grid‑world benchmark and demonstrate scalability on two challenging continuous‑control tasks.
By Florian Wolf, Ilyas Fatkhullin, Niao He
The paper introduces Solver-Gradient Guided Reinforcement Learning (SG‑RL), a method that augments standard RL with bounded gradients from a differentiable MPC solver to adapt cost‑function weights online. SG‑RL integrates solver‑gradient guidance into PPO through actor‑update scaling, policy loss, advantage estimation, and value‑function learning, achieving comparable or superior closed‑loop performance while requiring up to 70.6% fewer samples. Experiments on two autonomous racing platforms with intentional model mismatch demonstrate that SG‑RL outperforms both RL and gradient‑based policy learning baselines and generalizes zero‑shot to unseen environments.
By Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, S\'ebastien Gros, Davide Scaramuzza, Johannes Betz
The paper introduces Network Feasibility Geometry Reinforcement Learning (NFG‑RL), a method that enforces multi‑layer network constraints—such as interference, power‑rate coupling, flow conservation, service chains, capacity, latency, and reliability—by transporting a proto‑policy through a differentiable feasibility map. By compiling heterogeneous constraints into typed residual blocks and using a variational transport operator, NFG‑RL ensures almost‑sure feasible execution and shapes exploration and gradients to respect active constraints. Experiments on two wireless‑edge surrogate environments show that NFG‑RL boosts feasible utility by 37.5–41.5 %, cuts raw‑action violations by 48.5–60.8 %, and reduces P99 delay by 57.0–75.5 % compared to leading baselines.
By Zuyuan Zhang, Zeyu Fang, Mahdi Imani, Nathaniel D. Bastian, Tian Lan
arXiv:2610.35748v1 Announce Type: new
Abstract: Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ra...
By Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus
In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance. The widely adopted family of approximations rel...
arXiv:2606. 16564v1 Announce Type: cross Abstract: Robotic systems routinely encounter conflicting objectives, modeling errors, and degenerate contact conditions that render quadratic programs (QPs) infeasible.
By Aristotelis Papatheodorou, Jose Rojas, Ioannis Havoutis, Carlos Mastalli
The paper presents the first provably efficient offline reinforcement learning algorithms for risk-sensitive settings using optimized certainty equivalents (OCE). It introduces two formulations—dynamic-OCE, applying OCE at each step, and static-OCE, applying a single OCE to cumulative rewards—and develops pessimistic value-iteration methods with tight sublinear suboptimality bounds in tabular and linear function approximation settings. The authors also establish minimax lower bounds, characterizing the fundamental limits of offline risk-sensitive RL.
By Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu
arXiv:2602.08054v2 Announce Type: replace-cross
Abstract: Offline reinforcement learning (RL) offers a promising framework for deploying autonomous systems in safety-critical settings without the ris...
By Manan Tayal, Mumuksh Tayal, Ravi Prakash
arXiv:2610.34208v1 Announce Type: cross
Abstract: Offline safe reinforcement learning (RL) learns safety-aware policies from static datasets while satisfying safety constraints during deployment. Mot...
By Changxu Ni, Feiran You, Yongan Guo, Hongyang Du, Kang Wei, Le Liang, Yiyang Ni, Jun Li
The paper introduces a model‑free continuous‑time reinforcement learning algorithm that learns from a fixed batch of discrete‑time off‑policy data. It builds on the PhiBE‑Q framework to create a PhiBE‑PI policy iteration that uses a single state‑action critic, avoiding explicit drift or diffusion estimation. The authors provide a local end‑to‑end suboptimality guarantee under linear function approximation, analyze error propagation from discretization, function approximation, and finite samples, and reveal a bias‑variance trade‑off where finer time discretization can worsen statistical error.
By Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu