arXiv Machine Learning

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

The paper introduces a continuous‑time policy gradient framework that works with discrete observation data. It derives performance difference lemmas and policy gradient formulas for randomized and parametrized policies, proving exponential convergence for an entropy‑regularized policy flow and providing explicit convergence bounds for discrete updates. By integrating with PhiBE, the authors create model‑free actor‑critic algorithms that maintain the continuous‑time structure, and demonstrate their effectiveness on both linear‑quadratic and nonlinear robotic control tasks.

arXiv Machine Learning

Sep 15

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

The paper investigates continuous‑time stochastic control problems with unknown drift and running reward functions, using an exploratory reinforcement learning framework that incorporates relaxed controls and entropy regularization. It develops policy‑iteration algorithms based on probabilistic representations of the optimal value function and its gradient, proving convergence and demonstrating performance through numerical examples. The study also extends to a special case with control‑dependent diffusion, requiring a Hessian representation.

By Jin Ma, Gaozhan Wang, Jianfeng Zhang, Xunyu Zhou

diffusion reinforcement-learning

arXiv Machine Learning

19h ago

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

arXiv:2610.28915v1 Announce Type: new Abstract: In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed of...

By Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu

diffusion reinforcement-learning safety

arXiv Machine Learning

Oct 12

Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

The paper introduces an operator-theoretic approach to offline reinforcement learning for continuous-time stochastic systems. By linking reinforcement learning to the Hamilton‑Jacobi‑Bellman equation, it proposes a dynamic programming recursion that represents the world model through the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. The authors establish global convergence of the value function under small generator‑approximation error and provide explicit finite‑sample guarantees tied to system properties such as smoothness and stability.

By Nicolas Hoischen, Petar Bevanda, Max Beier, Stefan Sosnowski, Boris Houska, Sandra Hirche

agents diffusion reinforcement-learning robotics

arXiv Machine Learning

Sep 21

Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning

arXiv:2505.04193v2 Announce Type: replace Abstract: Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive resu...

By Bang You, Chenxu Wang, Wenju Yang, Di Guo, Huaping Liu

agents reinforcement-learning robotics benchmarks safety

arXiv AI

Jun 10

Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

arXiv:2606. 11087v1 Announce Type: cross Abstract: Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control.

By Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, Sergey Levine

diffusion reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 9

Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertainty

arXiv:2404. 13879v5 Announce Type: replace Abstract: Uncertainties in transition dynamics pose a critical challenge in reinforcement learning (RL), often resulting in performance degradation of trained policies when deployed on hardware.

By Xulin Chen, Ruipeng Liu, Zhenyu Gan, Garrett E. Katz

reinforcement-learning robotics safety

arXiv Machine Learning

Jul 9

Mathematical methods of reinforcement learning

arXiv:2607. 06935v1 Announce Type: cross Abstract: Reinforcement learning (RL) is increasingly grounded in tools from probability, optimization, and operator theory.

By Denis Belomestny, Alexander Gasnikov, Egor Gladin, Alexey Naumov, Artemy Rubtsov, Yuri Sapronov, Daniil Tiapkin, Nikita Yudin

reinforcement-learning

arXiv Machine Learning

Jun 26

Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data

arXiv:2606. 26498v1 Announce Type: cross Abstract: This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available.

By Erhan Bayraktar, Martin Hernandez, Qinxin Yan, Yuhua Zhu

diffusion reinforcement-learning benchmarks

arXiv AI

Jun 6

Retry Policy Gradients in Continuous Action Spaces

arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.

By Soichiro Nishimori, Paavo Parmas

arXiv Machine Learning

19h ago

A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment

arXiv:2610.29187v1 Announce Type: cross Abstract: Motivated by ten years of Learning-Based Control Invited Sessions at the IEEE Conference on Decision and Control (CDC), this tutorial paper reviews k...

By Angela P. Schoellig, Matthias A. M\"uller, Sebastian Trimpe, Melanie N. Zeilinger, Mahathi Anand, Andrea Carron, Oliver Hausd\"orfer, Victor G. Lopez, Friedrich Solowjow, Haocheng Zhao, Siqi Zhou

agents reinforcement-learning safety

Hugging Face Trending Papers

Sep 23

Limiting-Kernel Q($λ$): Bridging Short and Long Horizons

In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance. The widely adopted family of approximations rel...

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

Trajectory Data Suffices for Statistically Efficient Policy Evaluation in Fixed-Horizon Offline RL with Linear $q^\pi$-Realizability and Concentrability

arXiv:2510. 03494v2 Announce Type: replace Abstract: We study finite-horizon offline reinforcement learning (RL) with function approximation for both policy evaluation and policy optimization.

By Volodymyr Tkachuk, Csaba Szepesv\'ari, Xiaoqi Tan

reinforcement-learning

Related stories