← Back to all news
arXiv Machine Learning By Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, Hongxia Yang

InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Read the original on arXiv Machine Learning →

InfiFPO is a new preference optimization technique for implicit model fusion that replaces the reference model in Direct Preference Optimization with a fused source model, preserving probability information and avoiding complex vocabulary alignment. By incorporating probability clipping and max‑margin fusion, it aligns a pivot model with human preferences while distilling knowledge from multiple source models. Experiments on 11 benchmarks show that InfiFPO consistently outperforms existing fusion and preference optimization methods, raising Phi‑4’s average score from 79.95 to 83.33 and improving performance in mathematics, coding, and reasoning tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 26

Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

The paper introduces BALIGN, a balanced data selection strategy designed to reduce catastrophic forgetting—referred to as the alignment tax—in large language models during preference-based alignment. By analyzing preference optimization gradients, the authors identify three data-centric features that influence parameter drift: the reference model's log-probability margin, token length differences between chosen and rejected responses, and TF‑IDF similarity to general capability corpora. BALIGN aggregates these features into a composite risk score to filter out high-risk preference samples, thereby preserving foundational capabilities while maintaining alignment gains with minimal computational overhead.

By Minsu Kim, Jianxun Lian, Xing Xie, Steven Euijong Whang
llmssafety
More like this →