arXiv Machine Learning September 17, 2026 By Yury Kolomeytsev

A Convergence Framework for Deep $V$-Learning: Error Propagation and Sharp Action-Gap Bounds

Read the original on arXiv Machine Learning →

The paper presents a convergence framework for deep $V$‑learning over a finite horizon $H$, deriving explicit bounds on policy loss by decomposing the Bellman update error into six residuals. It shows how $L^s$ concentrability controls expected $L^1$ loss, quantifies the impact of shared sampling across horizon levels, and provides optimal and near‑optimal sample allocations for statistical error rates. The work also establishes sharp action‑gap bounds under a margin condition, transfers optimal‑gap results to frozen‑iterate gaps, and offers consistency guarantees for generative‑reset approximate‑ERM procedures with exact action scores.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Oct 2

Fast Regularized Policy Mirror Descent with One-Step TD Updates

The paper introduces Fast Regularized Policy Mirror Descent (PMD) that pairs policy updates with a single temporal-difference (TD) critic step. It proves global linear convergence for finite discounted MDPs using exact coordinate-wise Bellman updates and any positive actor stepsize, regardless of critic initialization. For stochastic TD-PMD with strongly convex mirror maps, the authors achieve an expected value gap of ε after “~O(1/((1-γ)^5 σ_b ε))” transitions, without requiring trajectory resets, generative models, or nested policy evaluation loops.

By Qipei Chen, Wenye Li, Yule Sun, Ke Wei

reinforcement-learning

arXiv Machine Learning

1d ago

Near-Minimax Sample Complexity for Multi-Objective Reinforcement Learning under Generative Models

arXiv:2611.05562v1 Announce Type: cross Abstract: In multi-objective reinforcement learning, preferences may be specified only after data collection, requiring one dataset to support a continuum of r...

By Yue Wang

reinforcement-learning

arXiv Statistics ML

Oct 26

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

arXiv:2610.32125v1 Announce Type: new Abstract: Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function $Q^*$ from a fixed dataset without further in...

By Hao Cheng, Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Chen Zhong

diffusion reinforcement-learning

arXiv Machine Learning

Sep 30

Optimal Multi-Reward Reinforcement Learning

arXiv:2609.36486v1 Announce Type: new Abstract: We study an unknown-transition finite-horizon Markov decision process (MDP) with a finite collection of known reward functions $\{r^1, r^2, \ldots, r^M...

By Zijun Chen, Zihan Zhang

reinforcement-learning

arXiv AI

Jun 29

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.

By Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, Baoxiang Wang

llms reinforcement-learning

arXiv Statistics ML

Oct 27

Minimax Regret for Sobolev Kernel-Based Reinforcement Learning

arXiv:2610.34045v1 Announce Type: cross Abstract: We study the minimax regret rate for episodic kernel-based reinforcement learning, where the expected reward and transition density belong to a Sobol...

By Haichen Hu, David Simchi-Levi

reinforcement-learning safety

One email a morning, machine-written

Related stories

Fast Regularized Policy Mirror Descent with One-Step TD Updates

Near-Minimax Sample Complexity for Multi-Objective Reinforcement Learning under Generative Models

Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Optimal Multi-Reward Reinforcement Learning

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

Minimax Regret for Sobolev Kernel-Based Reinforcement Learning