Efficient Adjoint Matching (EAM) is a new method for reward fine‑tuning of pretrained diffusion models that reduces computational cost by reformulating the stochastic optimal control problem with a linear base drift and a modified terminal cost. This change eliminates the need for extensive stochastic simulation of full generative trajectories and backward ODE simulation of the adjoint state, allowing training‑time sampling with a few‑step deterministic ODE solver and a closed‑form adjoint solution. On standard text‑to‑image reward fine‑tuning benchmarks, EAM converges up to four times faster than the previous Adjoint Matching approach while matching or surpassing it on metrics such as PickScore, ImageReward, HPSv2.1, CLIPScore, and Aesthetics.
By Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, Jaemoo Choi
arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.
By Boshu Lei, Kostas Daniilidis, Antonio Loquercio
The paper introduces Q‑Learning with Scalar Adjoint Matching (SQAM), a method that replaces expensive per‑step vector–Jacobian products in flow policy fine‑tuning with a closed‑form scalar adjoint that scales the value gradient by flow time. SQAM also incorporates a value penalty on policy‑generated actions, which is shown to be crucial for effective learning. Experiments demonstrate that SQAM outperforms strong baselines on the hardest OGBench domains and improves supervised fine‑tuning on a real bimanual robot with a vision‑language‑action policy.
By Yonghoon Dong, Minsung Yoon, Jaehyuk Kim, Jungwoo Park, Changyeon Kim, Jinwoo Shin
The paper introduces Wasserstein‑Tilted Flow Maps (WTF), a simulation‑free reinforcement learning method that fine‑tunes pre‑trained flow‑based generative models by adding an optimal transport regularizer derived from the model’s drift. Unlike traditional KL‑reward tilting, WTF transports individual samples toward higher reward, framing the problem as a deterministic optimal control task on the flow map. Experiments on ImageNet‑256 and text‑to‑image demonstrate that WTF achieves higher reward and comparable or better diversity while reducing training compute by up to 280×.
By Abbas Mammadov, Jerry Y. Huang, Justin Lin, Partha Kaushik, Sheel Shah, Kartik Nair, Yee Whye Teh, Nicholas M. Boffi
The paper introduces Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), a reinforcement learning framework that uses ODE-based flow matching to overcome limitations of SDE-based diffusion policies. FMER derives an analytic entropy objective to correct density distortion from a tanh transformation, establishes that minimizing an advantage-weighted conditional flow matching loss serves as a simulation-free surrogate for policy mirror descent, and dynamically tunes the mirror descent temperature to enforce a robust trust region. Experiments show FMER outperforms baselines on the sparse-reward FrankaKitchen task while remaining competitive on dense-reward MuJoCo benchmarks.
By Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi
Fenchel Tilt Flow Control (FTFC) is a new method for fine‑tuning pretrained generative models to arbitrary preference functions. It decouples utility optimization from model fitting by first learning reward and density‑ratio weights on pretrained samples, then freezing these weights to adjust a diffusion or flow model in a single importance‑weighted stage. The approach supports general f‑divergence penalties, achieves exact duality for concave utilities, and demonstrates up to 20× efficiency gains while outperforming baselines on image and molecule generation tasks.
By Maksim Bobrin, Maksim Zhdanov, Dmitry Dylov
arXiv:2509.25050v2 Announce Type: replace
Abstract: Reinforcement Learning (RL) has emerged as a central paradigm for advancing Large Language Models (LLMs), where both pre-training and RL post-train...
By Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, Zhi-Ming Ma
arXiv:2608.23664v1 Announce Type: cross
Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing method...
By Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen
arXiv:2606. 19162v1 Announce Type: new Abstract: Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself.
By Nicolas Beltran-Velez, Felix Friedrich, Zhang Xiaofeng, Reyhane Askari-Hemmat, Xiaochuang Han, Adriana Romero-Soriano, Michal Drozdzal
arXiv:2604. 17415v3 Announce Type: replace-cross Abstract: Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model.
By Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye
arXiv:2610.02260v1 Announce Type: new
Abstract: Flow matching models excel at generative modeling, and many downstream applications require their samples to satisfy prescribed constraints, such as ob...
By Yesom Park, Kelvin Kan, Qifan Chen, Thomas Flynn, Hayden Schaeffer. Xihaier Luo
arXiv:2605.12379v2 Announce Type: replace-cross
Abstract: Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching a...
By Fairoz Nower Khan, Nabuat Zaman Nahim, Peizhong Ju