arXiv AI

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

The paper introduces Diffusion-Augmented Markov Decision Processes (DA‑MDPs), a framework that extends Maximum Entropy Reinforcement Learning to diffusion-based policies. DA‑MDPs treat each reverse‑diffusion step as an RL decision, deriving a tractable reverse‑KL bound that decomposes across denoising transitions and yields diffusion‑augmented soft rewards, value functions, and policy objectives. The authors implement this framework with PPO, REPPO, and a maximum‑entropy WPO variant, showing improved continuous‑control performance, higher success rates on manipulation tasks, and memory‑efficient training with action chunking.

arXiv Machine Learning
Jul 17

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models

arXiv:2607. 14522v1 Announce Type: new Abstract: We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynamics are modeled as a controlled continuous-time Markov chain (CTMC).

By Zikun Zhang, Jiayuan Sheng, David D. Yao, Wenpin Tang
arXiv Machine Learning
Jul 20

Dichotomous Diffusion Policy Optimization

arXiv:2601. 00898v3 Announce Type: replace Abstract: Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference.

By Ruiming Liang, Yinan Zheng, Kexin Zheng, Tianyi Tan, Jianxiong Li, Liyuan Mao, Zhihao Wang, Guang Chen, Hangjun Ye, Jingjing Liu, Jinqiao Wang, Xianyuan Zhan
arXiv AI
Oct 16

Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

The paper reviews how reinforcement learning (RL) can refine diffusion models (DMs) after pretraining, addressing limitations in reasoning, exploration, and alignment. It surveys RL-driven DMs across language, multimodal, robotic, and scientific domains, focusing on trajectory optimization and reward engineering. Experiments highlight the benefits of various RL post‑training strategies and identify key bottlenecks—reward fragility, exploration‑exploitation trade‑offs, and training instability—while proposing a roadmap for future research and a public repository for updates.

By Zhiyuan Ma, Jiabao Wei, Tianyu Wang, Hanqing Wang, Jiadong Pan, Haowen Xu, Liangliang Zhao, Ruixun Liu, Yuhan Wang, Yuwei Zhang, Liang Feng, Junhao Xiao, Kaiyan Zhang, Xuekai Zhu, Daizong Liu, Biqing Qi, Yihao Liu, Ganqu Cui, Ning Ding, Linfeng Zhang, Jianjun Li, Kun He, Xiang Bai, Bowen Zhou
arXiv Machine Learning
Oct 22

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

The paper introduces Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), a reinforcement learning framework that uses ODE-based flow matching to overcome limitations of SDE-based diffusion policies. FMER derives an analytic entropy objective to correct density distortion from a tanh transformation, establishes that minimizing an advantage-weighted conditional flow matching loss serves as a simulation-free surrogate for policy mirror descent, and dynamically tunes the mirror descent temperature to enforce a robust trust region. Experiments show FMER outperforms baselines on the sparse-reward FrankaKitchen task while remaining competitive on dense-reward MuJoCo benchmarks.

By Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi
arXiv AI
Nov 6

Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging

Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging introduces the Latent Noise Linear Markov Decision Process (LNL‑MDP) to address the noise aliasing problem in fine‑tuning pre‑trained diffusion and flow policies for robotics. By deriving a Bridge Equation that aligns noise and action spaces through a shared representation, the authors provide a framework that falls under FineTuneRL theory with near‑optimal guarantees. The resulting LARBRID algorithm learns this representation to optimize the initial noise distribution, achieving 2–3× performance gains on a variety of robotic locomotion and manipulation tasks.

By Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati