arXiv AI

Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models

Fenchel Tilt Flow Control (FTFC) is a new method for fine‑tuning pretrained generative models to arbitrary preference functions. It decouples utility optimization from model fitting by first learning reward and density‑ratio weights on pretrained samples, then freezing these weights to adjust a diffusion or flow model in a single importance‑weighted stage. The approach supports general f‑divergence penalties, achieves exact duality for concave utilities, and demonstrates up to 20× efficiency gains while outperforming baselines on image and molecule generation tasks.

arXiv Machine Learning

Sep 24

WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps

The paper introduces Wasserstein‑Tilted Flow Maps (WTF), a simulation‑free reinforcement learning method that fine‑tunes pre‑trained flow‑based generative models by adding an optimal transport regularizer derived from the model’s drift. Unlike traditional KL‑reward tilting, WTF transports individual samples toward higher reward, framing the problem as a deterministic optimal control task on the flow map. Experiments on ImageNet‑256 and text‑to‑image demonstrate that WTF achieves higher reward and comparable or better diversity while reducing training compute by up to 280×.

By Abbas Mammadov, Jerry Y. Huang, Justin Lin, Partha Kaushik, Sheel Shah, Kartik Nair, Yee Whye Teh, Nicholas M. Boffi

diffusion reinforcement-learning fine-tuning efficiency

arXiv Machine Learning

Jun 10

Exploring the Design Space of Reward Backpropagation for Flow Matching

arXiv:2606. 11075v1 Announce Type: new Abstract: Aligning text-to-image flow matching models with human preferences via direct reward backpropagation is sample-efficient but hampered by two well-known pathologies: activations cannot be stored across the full sampling trajectory at modern model scale, and chained Jacobian products across steps inflate the reward gradient as it travels back to early indices.

By Ruoyu Wang, Boye Niu, Xiangxin Zhou, Yushi Huang, Tongliang Liu, Chi Zhang

diffusion

arXiv AI

Jun 2

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

arXiv:2604. 17415v3 Announce Type: replace-cross Abstract: Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model.

By Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye

diffusion fine-tuning safety

arXiv Machine Learning

Sep 2

Elite-Weighted Supervised Fine-tuning for Goal-Directed Molecular Optimization

arXiv:2609.00189v1 Announce Type: new Abstract: Goal-directed optimization is essential for steering molecular generators to propose candidates with desired properties. However, it is often implement...

By Shiyun Wa, Yifei Wang, Anna G. Green, Simone Sciabola, Ye Wang

diffusion reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Sep 1

Reward-guided Fine-Tuning of One-Step Generative Models via Wasserstein Gradient Flow

arXiv:2608.29647v1 Announce Type: new Abstract: To mitigate the time complexity of generative models, one-step generative models have recently emerged through direct mapping from noise to data in a s...

By Hoseong Hwang, Woorim Han, Joungin Chun, Jinseong Park, Jaewoong Choi

fine-tuning safety

arXiv Machine Learning

Jun 19

Meta Flow Maps enable scalable reward alignment

arXiv:2601. 14430v2 Announce Type: replace-cross Abstract: Controlling generative models is computationally expensive.

By Peter Potaptchik, Adhi Saravanan, Abbas Mammadov, Alvaro Prat, Michael S. Albergo, Yee Whye Teh

fine-tuning safety

arXiv Machine Learning

Jul 3

Optimizing Visual Generative Models via Distribution-wise Rewards

arXiv:2607. 02291v1 Announce Type: new Abstract: Conventional reinforcement learning strategies for visual generation typically employ sample-wise reward functions, yet this practice frequently results in reward hacking that degrades image diversity and introduces visual anomalies.

By Ruihang Li, Mengde Xu, Shuyang Gu, Leigang Qu, Fuli Feng, Han Hu, Wenjie Wang

reinforcement-learning safety

arXiv Machine Learning

Jun 18

The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

arXiv:2606. 19162v1 Announce Type: new Abstract: Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself.

By Nicolas Beltran-Velez, Felix Friedrich, Zhang Xiaofeng, Reyhane Askari-Hemmat, Xiaochuang Han, Adriana Romero-Soriano, Michal Drozdzal

diffusion reinforcement-learning safety

arXiv Computer Vision

4d ago

Aligning One-Step Generative Models with Reward-Weighted Transport Distillation

arXiv:2609.30840v1 Announce Type: cross Abstract: One-step generators enable high-quality visual generation with a single network evaluation, but their post-training is difficult: general implicit ge...

By Austin Wang, Ziheng Cheng, Lexing Ying

efficiency safety

arXiv AI

Sep 10

Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling

arXiv:2609.05727v1 Announce Type: cross Abstract: We develop Newton Matching, a unified framework for fine-tuning and sampling in generative modeling. The target is $\pi\propto\mu e^{\tau r}$, where...

By Zeyang Li, Yunan Wang, Paolo Giaretta, Navid Azizan

reinforcement-learning fine-tuning

arXiv Machine Learning

Jun 17

Gradual Fine-Tuning for Flow Matching Models

arXiv:2601. 22495v2 Announce Type: replace Abstract: Fine-tuning flow matching models is a central challenge in settings with limited data, evolving distributions, or computational constraints.

By Gudrun Thorkelsdottir, Arindam Banerjee

diffusion fine-tuning

arXiv AI

Jun 2

Efficient Weighted Sampling via Score-based Generative Models

arXiv:2502. 04646v2 Announce Type: replace-cross Abstract: Weighted sampling -- sampling from a probability density function (PDF) proportional to the product of a base PDF and a weight function -- is a fundamental technique with wide-ranging applications in variance reduction, biased sampling, data augmentation, and more.

By Heasung Kim, Taekyun Lee, Hyeji Kim, Gustavo de Veciana

diffusion benchmarks

Related stories