The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.
By Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin
arXiv:2509.25050v2 Announce Type: replace
Abstract: Reinforcement Learning (RL) has emerged as a central paradigm for advancing Large Language Models (LLMs), where both pre-training and RL post-train...
By Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, Zhi-Ming Ma
arXiv:2606. 30376v1 Announce Type: new Abstract: Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods.
By Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu
arXiv:2609.06893v1 Announce Type: cross
Abstract: Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational eff...
By Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi
The paper investigates when conditional flow matching (CFM) can replace pointwise negative log-likelihood (NLL) calculations. It shows that for linear Gaussian paths, the endpoint NLL can be exactly decomposed into entropy, a weighted CFM objective, and residual terms, meaning CFM-only estimates are exact only when these residuals cancel. The study finds that ordinary CFM is generally not a pointwise NLL estimator, and even weighted variants may not fully eliminate bias, especially in training or on‑policy settings, with experiments confirming these theoretical insights.
By Yansen Han, Hongxin Sun, Tao Lin
arXiv:2610.16275v1 Announce Type: cross
Abstract: Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling express...
By Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju
arXiv:2608. 14430v1 Announce Type: new Abstract: Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards.
By Yixian Xu, Yuanrui Zhang, Shengjie Luo, Liwei Wang, Di He
arXiv:2606. 02521v1 Announce Type: new Abstract: One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization.
By Zhou Jiang, Yandong Wen, Zhen Liu
The paper introduces Cumulative Marginal Matching (CMM), a new objective for training few‑step diffusion and flow‑matching models. CMM treats flow‑matching as a sequence of intermediate marginal distributions and directly optimizes the cumulative Wasserstein‑2 discrepancy between the base and few‑step models across the entire trajectory. It derives a tractable surrogate for this objective, shows that Shortcut‑CSL is a first‑order approximation of CMM, and reports improved generation quality on standard benchmarks under the same training budget.
By Paribesh Regmi, Rui Li
arXiv:2510. 09976v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models such as OpenVLA, Octo, and $\pi_0$ have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data.
By Mingyang Lyu, Yinqian Sun, Erliang Lin, Huangrui Li, Ruolin Chen, Feifei Zhao, Yi Zeng
arXiv:2604. 17415v3 Announce Type: replace-cross Abstract: Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model.
By Jeongjae Lee, Jinho Chang, Jeongsol Kim, Jong Chul Ye
arXiv:2609.36812v1 Announce Type: cross
Abstract: Diffusion and flow policies can model complex behaviors in offline reinforcement learning (RL). However, penalizing their KL divergence from the beha...
By Junhyun Ha, Juho Lee, Byungwoo Park