The paper introduces a continuous‑time policy gradient framework that works with discrete observation data. It derives performance difference lemmas and policy gradient formulas for randomized and parametrized policies, proving exponential convergence for an entropy‑regularized policy flow and providing explicit convergence bounds for discrete updates. By integrating with PhiBE, the authors create model‑free actor‑critic algorithms that maintain the continuous‑time structure, and demonstrate their effectiveness on both linear‑quadratic and nonlinear robotic control tasks.
By Martin Hernandez, Xinyan Qing, Mo Zhou, Yuhua Zhu
The paper introduces a model‑free continuous‑time reinforcement learning algorithm that learns from a fixed batch of discrete‑time off‑policy data. It builds on the PhiBE‑Q framework to create a PhiBE‑PI policy iteration that uses a single state‑action critic, avoiding explicit drift or diffusion estimation. The authors provide a local end‑to‑end suboptimality guarantee under linear function approximation, analyze error propagation from discretization, function approximation, and finite samples, and reveal a bias‑variance trade‑off where finer time discretization can worsen statistical error.
By Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu
arXiv:2606. 28671v1 Announce Type: new Abstract: Stackelberg differential games (SDGs) provide a powerful framework for hierarchical decision-making in stochastic and continuous-time environments, yet their solution remains computationally challenging due to the complexity of traditional dynamic programming and Hamilton-Jacobi-Bellman-Isaacs (HJBI) methods, especially in high-dimensional systems.
By Congde Hu, Danping Li, Lin Xu, Wenying Xu
arXiv:2606. 05888v1 Announce Type: new Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses.
By Soichiro Nishimori, Paavo Parmas
The paper introduces reinforcement learning for Continuous-Time Jump Markov Decision Processes (CTJMDPs) with general discrete state spaces and continuous/discrete actions. It develops entropy‑regularized continuous‑time control and establishes theoretical foundations for q‑learning in this setting, providing model‑free algorithms that outperform naive discretization. Numerical tests on network dynamic pricing demonstrate the method’s ability to learn near‑optimal policies and scale to large networks.
By Huiling Meng, Ningyuan Chen, Xuefeng Gao
arXiv:2506. 08121v2 Announce Type: replace-cross Abstract: We introduce a continuous policy-value iteration algorithm where the approximations of the value function of a stochastic control problem and the optimal control are simultaneously updated through Langevin-type dynamics.
By Qi Feng, Gu Wang
arXiv:2606. 28669v1 Announce Type: new Abstract: To address parameter misspecification and sudden structural environmental changes in conventional stochastic differential game (SDG) frameworks, this paper introduces a distributional control approach that characterizes optimal strategies as probability distributions over actions, conditioned on the continuous state, the discrete regime state, and parameters.
By Congde Hu, Zhuo Jin, Danping Li, Lin Xu
arXiv:2609.06882v1 Announce Type: cross
Abstract: Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remain...
By Mahmoud Selim, Cristina Cipriani, Karl H. Johansson
The paper introduces BUMEX, a reinforcement learning exploration strategy that leverages a set of prior models containing the true transition kernel and reward function. By optimizing over this model set, the method derives upper and lower bounds on the Q‑function to guide exploration, providing theoretical guarantees of convergence to the optimal policy. When the model set follows a bounded‑parameter MDP structure, the optimization becomes convex, enabling finite‑time convergence under mild assumptions and demonstrating accelerated learning in simulations.
By J. S. van Hulst, W. P. M. H. Heemels, D. J. Antunes
The paper introduces Diffusion-Augmented Markov Decision Processes (DA‑MDPs), a framework that extends Maximum Entropy Reinforcement Learning to diffusion-based policies. DA‑MDPs treat each reverse‑diffusion step as an RL decision, deriving a tractable reverse‑KL bound that decomposes across denoising transitions and yields diffusion‑augmented soft rewards, value functions, and policy objectives. The authors implement this framework with PPO, REPPO, and a maximum‑entropy WPO variant, showing improved continuous‑control performance, higher success rates on manipulation tasks, and memory‑efficient training with action chunking.
By Sebastian Sanokowski, Kaustubh Patil, Majid Khadiv
arXiv:2605. 14982v2 Announce Type: replace-cross Abstract: We address the discounted reward setting in reinforcement learning (RL).
By Sanjeev Manivannan, Shuban V
arXiv:2411. 01302v2 Announce Type: replace Abstract: We study the convergence of $q$-learning and related algorithms introduced by Jia and Zhou (J.
By Wenpin Tang, Xun Yu Zhou