arXiv AI By Yansen Han, Shengyi Liao, Yuanxing Zhang, Pengfei Wan, Tao Lin

Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

Read the original on arXiv AI →

arXiv:2608. 20011v1 Announce Type: new Abstract: Preference optimization is a standard alignment method for generative models, yet extending it to continuous-time dynamics remains non-trivial.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Jun 10

Exploring the Design Space of Reward Backpropagation for Flow Matching

arXiv:2606. 11075v1 Announce Type: new Abstract: Aligning text-to-image flow matching models with human preferences via direct reward backpropagation is sample-efficient but hampered by two well-known pathologies: activations cannot be stored across the full sampling trajectory at modern model scale, and chained Jacobian products across steps inflate the reward gradient as it travels back to early indices.

By Ruoyu Wang, Boye Niu, Xiangxin Zhou, Yushi Huang, Tongliang Liu, Chi Zhang