← Back to all news
arXiv Machine Learning By Defu Cao, Angela Zhou

Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions

Read the original on arXiv Machine Learning →

The paper introduces state abstractions that preserve the difference of Q‑functions for offline reinforcement learning, aiming to exclude irrelevant dynamics from rich state data. It proposes a dynamic generalization of the R‑learner that uses orthogonal estimation and sparse learning to estimate the Q‑function contrast, achieving faster convergence and consistency under a margin condition. Experiments on simulated and simulator‑augmented real data show variance reductions and demonstrate that the necessary information for sequential decision‑making can be smaller than that required for full state prediction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 22

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

The paper introduces a model‑free continuous‑time reinforcement learning algorithm that learns from a fixed batch of discrete‑time off‑policy data. It builds on the PhiBE‑Q framework to create a PhiBE‑PI policy iteration that uses a single state‑action critic, avoiding explicit drift or diffusion estimation. The authors provide a local end‑to‑end suboptimality guarantee under linear function approximation, analyze error propagation from discretization, function approximation, and finite samples, and reveal a bias‑variance trade‑off where finer time discretization can worsen statistical error.

By Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu
diffusionreinforcement-learningsafety
More like this →
arXiv Machine Learning
Sep 24

Limiting-Kernel Q($\lambda$): Bridging Short and Long Horizons

Limiting‑Kernel Q(λ) (LKQL) is an off‑policy value estimator that blends n‑step truncation with a long‑horizon approximation based on the limiting kernel. It maintains the computational efficiency of n‑step methods while improving policy evaluation accuracy, especially for long‑horizon tasks. The authors prove faster convergence of LKQL’s operator under aperiodicity and near‑on‑policy conditions, and demonstrate empirical gains on MuJoCo continuous‑control benchmarks.

By Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
reinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Aug 24

Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models

The paper introduces BUMEX, a reinforcement learning exploration strategy that leverages a set of prior models containing the true transition kernel and reward function. By optimizing over this model set, the method derives upper and lower bounds on the Q‑function to guide exploration, providing theoretical guarantees of convergence to the optimal policy. When the model set follows a bounded‑parameter MDP structure, the optimization becomes convex, enabling finite‑time convergence under mild assumptions and demonstrating accelerated learning in simulations.

By J. S. van Hulst, W. P. M. H. Heemels, D. J. Antunes
agentsreinforcement-learningbenchmarks
More like this →