arXiv AI October 12, 2026 By Xiang Li, Yuheng Zhang, Nan Jiang

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies

The paper extends theoretical guarantees for offline reinforcement learning to parameterized policy classes over large or continuous action spaces, overcoming limitations of prior algorithms that required finite action sets and state‑wise mirror descent. By addressing contextual coupling and linking mirror descent to natural policy gradients, the authors provide new analyses, algorithmic insights, and a surprising connection between offline RL and imitation learning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

reinforcement-learning

arXiv Machine Learning

Jun 2

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^\pi$-Realizability and Concentrability

arXiv:2510. 03494v2 Announce Type: replace Abstract: We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization.

By Volodymyr Tkachuk, Csaba Szepesv\'ari, Xiaoqi Tan

reinforcement-learning

arXiv Machine Learning

Aug 24

Decoupling Policy Extraction for Offline Reinforcement Learning

arXiv:2608.20909v1 Announce Type: new Abstract: Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled l...

By Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, Zongbao Feng, Peng Jia

rag reinforcement-learning

arXiv AI

Jul 17

Fully Offline Reinforcement Learning

arXiv:2505. 22442v3 Announce Type: replace-cross Abstract: Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance.

By Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel, Maike Osborne, Jakob N. Foerster

reinforcement-learning

arXiv AI

Jul 3

Generalization in offline RL: The structure is more important than the amount of pessimism

arXiv:2607. 02288v1 Announce Type: cross Abstract: While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization.

By Max Weltevrede, Matthijs T. J. Spaan, Wendelin B\"ohmer

reinforcement-learning safety

arXiv Machine Learning

Jun 18

When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?

arXiv:2606. 18531v1 Announce Type: cross Abstract: Offline reinforcement learning is typically analyzed under process-level reward supervision, yet many sequential decision datasets record only trajectory-level outcomes.

By Xuanfei Ren, Tengyang Xie

reinforcement-learning

arXiv AI

Aug 11

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

arXiv:2602. 17632v3 Announce Type: replace-cross Abstract: Modern offline Reinforcement Learning (RL) methods find performant actor-critics, however, fine-tuning these actor-critics online with value-based RL algorithms typically causes immediate drops in performance.

By Nathan Samuel de Lara, Florian Shkurti

reinforcement-learning fine-tuning

One email a morning, machine-written

Related stories

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^\pi$-Realizability and Concentrability

Decoupling Policy Extraction for Offline Reinforcement Learning

Fully Offline Reinforcement Learning

Generalization in offline RL: The structure is more important than the amount of pessimism

When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?

SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer