The paper introduces Diffusion-Augmented Markov Decision Processes (DA‑MDPs), a framework that extends Maximum Entropy Reinforcement Learning to diffusion-based policies. DA‑MDPs treat each reverse‑diffusion step as an RL decision, deriving a tractable reverse‑KL bound that decomposes across denoising transitions and yields diffusion‑augmented soft rewards, value functions, and policy objectives. The authors implement this framework with PPO, REPPO, and a maximum‑entropy WPO variant, showing improved continuous‑control performance, higher success rates on manipulation tasks, and memory‑efficient training with action chunking.
By Sebastian Sanokowski, Kaustubh Patil, Majid Khadiv
arXiv:2601. 00898v3 Announce Type: replace Abstract: Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference.
By Ruiming Liang, Yinan Zheng, Kexin Zheng, Tianyi Tan, Jianxiong Li, Liyuan Mao, Zhihao Wang, Guang Chen, Hangjun Ye, Jingjing Liu, Jinqiao Wang, Xianyuan Zhan
arXiv:2608. 20208v1 Announce Type: new Abstract: Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction.
By Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang, Sha Zhang, Jiajun Deng, Yanyong Zhang
arXiv:2607. 15273v1 Announce Type: cross Abstract: MeanFlow generators achieve fast few-step sampling by predicting average velocities over time intervals, making them attractive for efficient generation.
By Yushi Huang, Xiangxin Zhou, Jun Zhang, Liefeng Bo, Tianyu Pang
The book "The Principles of Diffusion Models" outlines the foundational concepts behind diffusion models, tracing their evolution from a forward process that corrupts data into noise to a reverse process that reconstructs data. It presents three complementary perspectives—variational, score-based, and flow-based—each describing how a time-dependent velocity field transports a simple prior to the data distribution. The text also covers practical guidance for controllable generation, efficient solvers, and diffusion-inspired flow-map models, providing a mathematically grounded framework for readers with basic deep‑learning knowledge.
By Chieh-Hsin Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji, Stefano Ermon
arXiv:2603. 27450v2 Announce Type: replace Abstract: Thanks to their remarkable flexibility, diffusion models and flow models have emerged as promising candidates for policy representation.
By Chenxiao Gao, Edward Chen, Tianyi Chen, Bo Dai
arXiv:2601. 22211v2 Announce Type: replace Abstract: Reinforcement learning (RL) with combinatorial action spaces remains challenging because feasible action sets are exponentially large and governed by complex feasibility constraints, making direct policy parameterization impractical.
By Lingkai Kong, Anagha Satish, Hezi Jiang, Akseli Kangaslahti, Andrew Ma, Wenbo Chen, Mingxiao Song, Lily Xu, Milind Tambe
arXiv:2606. 06967v1 Announce Type: new Abstract: Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning (RL) in complex continuous-control tasks.
By Ke Hu, Shutong Ding, Panxin Tao, Jingya Wang, Ye Shi
arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.
By Boshu Lei, Kostas Daniilidis, Antonio Loquercio
arXiv:2608. 07870v1 Announce Type: new Abstract: Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly.
By Donghu Kim, Youngdo Lee, Hojoon Lee, Johan Obando-Ceron, Byungkun Lee, Aaron Courville, Pablo Samuel Castro, Jaegul Choo, Clare Lyle
The paper introduces Bayesian Flow Networks for Offline Trajectory Planning (BFN-RL), a generative modeling framework that unifies discrete and continuous trajectory synthesis for offline reinforcement learning. Unlike prior diffusion models that rely on Gaussian noise, BFN-RL iteratively updates distribution parameters, enabling a categorical planner to produce future state sequences and an inverse-dynamics model to translate these states into actions. Experiments demonstrate that BFN-RL effectively generates trajectories in both discrete planning and continuous control tasks, highlighting its versatility across data modalities.
By Ludvig Killingberg, Helge Langseth
arXiv:2604. 14698v2 Announce Type: replace Abstract: Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL).
By Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng