arXiv Machine Learning

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion proposes a new inference-time framework that improves steering of frozen masked discrete diffusion models. By reducing the variance of guidance estimates, applying reward tilting to clean-token logits, and adapting the selection temperature at each step, VGAS addresses three default choices in existing pipelines. Experiments on regulatory DNA, protein, and small-molecule benchmarks show that VGAS achieves the best training-free reward performance and matches or surpasses reward-fine-tuned generators.

arXiv Machine Learning

Aug 28

GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

The paper introduces GRAS, a method that improves training‑free reward alignment for discrete diffusion models by reducing variance in guided proposals and adapting the resampling temperature during search. It achieves this without adding denoiser cost, using Rao‑Blackwellized estimates for differentiable rewards and a leave‑one‑out baseline for non‑differentiable ones. Experiments on regulatory DNA and protein design show GRAS outperforms existing training‑free techniques and rivals reward‑fine‑tuned models.

By Kwanyoung Kim

diffusion fine-tuning safety

arXiv Machine Learning

Jul 7

On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimization

arXiv:2607. 02834v1 Announce Type: new Abstract: Molecular optimization often starts from a pretrained generative model that captures a broad prior over valid molecular structures.

By Trevor Chen, Ariel Dai, Jason Yang, Riccardo De Santi, Daniel Khalil, Wenda Chu, Nate Gruver, Pranav Murugan, Alexander F. G. Goldberg, Maruan Al-Shedivat, Yisong Yue

diffusion fine-tuning

arXiv Machine Learning

Aug 5

Latent Reward Registers for Diffusion Preference Alignment

arXiv:2608. 03929v1 Announce Type: new Abstract: Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process.

By Yuanshen Guan, Zipeng Feng, Zhiwei Xiong, Peiqin Sun

llms diffusion reinforcement-learning efficiency benchmarks safety

arXiv AI

Jul 7

Trust-Region Noise Search for Black-Box Alignment of Diffusion and Flow Models

arXiv:2603. 14504v2 Announce Type: replace-cross Abstract: Optimizing the noise samples of diffusion and flow models is an increasingly popular approach to align these models to target rewards at inference time.

By Niklas Schweiger, Daniel Cremers, Karnik Ram

diffusion safety

arXiv Computer Vision

4d ago

Aligning One-Step Generative Models with Reward-Weighted Transport Distillation

arXiv:2609.30840v1 Announce Type: cross Abstract: One-step generators enable high-quality visual generation with a single network evaluation, but their post-training is difficult: general implicit ge...

By Austin Wang, Ziheng Cheng, Lexing Ying

efficiency safety

arXiv AI

Jun 2

Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

arXiv:2604. 17415v3 Announce Type: replace-cross Abstract: Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model.

By Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye

diffusion fine-tuning safety

arXiv AI

4d ago

Spectral Feedback for Test-Time Alignment of Protein Diffusion Models

Spectral Feedback is a new algorithm for aligning discrete diffusion models at test time by iteratively revisiting and editing token positions rather than only steering the reverse process. It selects edit-sets—groups of token positions to re-mask and re-sample—using sparse Fourier representations of edit-set value functions, enabling efficient optimization of which tokens to revisit. The method is model-agnostic and improves alignment performance across pretrained, test‑time aligned, and fine‑tuned diffusion models, achieving significant gains in protein stability for inverse folding tasks.

By Shai Dickman, Mert Cemri, Landon Butler, Kannan Ramchandran

diffusion reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 2

Drifting Preference Optimization for One-Step Generative Models

arXiv:2606. 02521v1 Announce Type: new Abstract: One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization.

By Zhou Jiang, Yandong Wen, Zhen Liu

diffusion fine-tuning benchmarks safety

arXiv AI

Jul 13

Reward Transport: Property Control in Flow Matching via Noise-Space Alignment

arXiv:2607. 08781v1 Announce Type: cross Abstract: The coupling in flow matching -- the rule pairing noise vectors with data points -- is typically treated as a computational choice.

By Kehan Guo, Yili Shen, Yujun Zhou, Yue Huang, Chujie Gao, Shiyi Du, Xiangliang Zhang

diffusion reinforcement-learning safety

arXiv Machine Learning

Aug 26

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

arXiv:2608.23664v1 Announce Type: cross Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing method...

By Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen

llms diffusion reinforcement-learning fine-tuning

arXiv AI

1d ago

Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models

Fenchel Tilt Flow Control (FTFC) is a new method for fine‑tuning pretrained generative models to arbitrary preference functions. It decouples utility optimization from model fitting by first learning reward and density‑ratio weights on pretrained samples, then freezing these weights to adjust a diffusion or flow model in a single importance‑weighted stage. The approach supports general f‑divergence penalties, achieves exact duality for concave utilities, and demonstrates up to 20× efficiency gains while outperforming baselines on image and molecule generation tasks.

By Maksim Bobrin, Maksim Zhdanov, Dmitry Dylov

diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 18

Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization

arXiv:2606. 18961v1 Announce Type: new Abstract: Protein language models (PLMs) have emerged as powerful tools for controllable biomolecular design, yet their post-training adaptation typically relies on costly wet-lab validation or curated preference datasets.

By Lanqing Li, Shentong Mo, Yang Yu, Pheng-Ann Heng

llms reinforcement-learning fine-tuning

Related stories