arXiv Computer Vision
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv AI
1d ago
A Zeroth-Order Paradigm for LLM Preference Alignment
The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.
arXiv Machine Learning
Sep 2
Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
arXiv:2509.25050v2 Announce Type: replace Abstract: Reinforcement Learning (RL) has emerged as a central paradigm for advancing Large Language Models (LLMs), where both pre-training and RL post-train...
arXiv Machine Learning
Jun 30
FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification
arXiv:2606. 30376v1 Announce Type: new Abstract: Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods.
arXiv Machine Learning
Sep 10
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
arXiv:2609.06893v1 Announce Type: cross Abstract: Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational eff...
arXiv Machine Learning
Aug 31
When Can Conditional Flow Matching Replace Pointwise Negative Log-Likelihood?
The paper investigates when conditional flow matching (CFM) can replace pointwise negative log-likelihood (NLL) calculations. It shows that for linear Gaussian paths, the endpoint NLL can be exactly decomposed into entropy, a weighted CFM objective, and residual terms, meaning CFM-only estimates are exact only when these residuals cancel. The study finds that ordinary CFM is generally not a pointwise NLL estimator, and even weighted variants may not fully eliminate bias, especially in training or on‑policy settings, with experiments confirming these theoretical insights.
arXiv Machine Learning
Aug 17
Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
arXiv:2608. 14430v1 Announce Type: new Abstract: Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards.