arXiv Machine Learning
Weighted Adjoint Matching: Finetuning Flow Matching Models with Time-Dependent Control Cost
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 23
Efficient Adjoint Matching for Fine-tuning Diffusion Models
Efficient Adjoint Matching (EAM) is a new method for reward fine‑tuning of pretrained diffusion models that reduces computational cost by reformulating the stochastic optimal control problem with a linear base drift and a modified terminal cost. This change eliminates the need for extensive stochastic simulation of full generative trajectories and backward ODE simulation of the adjoint state, allowing training‑time sampling with a few‑step deterministic ODE solver and a closed‑form adjoint solution. On standard text‑to‑image reward fine‑tuning benchmarks, EAM converges up to four times faster than the previous Adjoint Matching approach while matching or surpassing it on metrics such as PickScore, ImageReward, HPSv2.1, CLIPScore, and Aesthetics.
arXiv AI
Jun 9
Reinforcement Learning for Flow-Matching Policies with Density Transport
arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.
arXiv Machine Learning
Oct 8
Q-Learning with Scalar Adjoint Matching
The paper introduces Q‑Learning with Scalar Adjoint Matching (SQAM), a method that replaces expensive per‑step vector–Jacobian products in flow policy fine‑tuning with a closed‑form scalar adjoint that scales the value gradient by flow time. SQAM also incorporates a value penalty on policy‑generated actions, which is shown to be crucial for effective learning. Experiments demonstrate that SQAM outperforms strong baselines on the hardest OGBench domains and improves supervised fine‑tuning on a real bimanual robot with a vision‑language‑action policy.
arXiv Machine Learning
Sep 24
WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps
The paper introduces Wasserstein‑Tilted Flow Maps (WTF), a simulation‑free reinforcement learning method that fine‑tunes pre‑trained flow‑based generative models by adding an optimal transport regularizer derived from the model’s drift. Unlike traditional KL‑reward tilting, WTF transports individual samples toward higher reward, framing the problem as a deterministic optimal control task on the flow map. Experiments on ImageNet‑256 and text‑to‑image demonstrate that WTF achieves higher reward and comparable or better diversity while reducing training compute by up to 280×.
arXiv Machine Learning
Oct 22
Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
The paper introduces Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), a reinforcement learning framework that uses ODE-based flow matching to overcome limitations of SDE-based diffusion policies. FMER derives an analytic entropy objective to correct density distortion from a tanh transformation, establishes that minimizing an advantage-weighted conditional flow matching loss serves as a simulation-free surrogate for policy mirror descent, and dynamically tunes the mirror descent temperature to enforce a robust trust region. Experiments show FMER outperforms baselines on the sparse-reward FrankaKitchen task while remaining competitive on dense-reward MuJoCo benchmarks.
arXiv AI
Oct 1
Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models
Fenchel Tilt Flow Control (FTFC) is a new method for fine‑tuning pretrained generative models to arbitrary preference functions. It decouples utility optimization from model fitting by first learning reward and density‑ratio weights on pretrained samples, then freezing these weights to adjust a diffusion or flow model in a single importance‑weighted stage. The approach supports general f‑divergence penalties, achieves exact duality for concave utilities, and demonstrates up to 20× efficiency gains while outperforming baselines on image and molecule generation tasks.