arXiv Machine Learning
Oct 22

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

The paper introduces Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), a reinforcement learning framework that uses ODE-based flow matching to overcome limitations of SDE-based diffusion policies. FMER derives an analytic entropy objective to correct density distortion from a tanh transformation, establishes that minimizing an advantage-weighted conditional flow matching loss serves as a simulation-free surrogate for policy mirror descent, and dynamically tunes the mirror descent temperature to enforce a robust trust region. Experiments show FMER outperforms baselines on the sparse-reward FrankaKitchen task while remaining competitive on dense-reward MuJoCo benchmarks.

By Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi