← Back to all news
arXiv AI By Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman

Learning Not To Interfere: Adapting Experts For Model Merging

Read the original on arXiv AI →

The paper introduces Resolving Interference (RI), a lightweight adaptation framework that reduces cross‑task interference when merging expert models. RI disentangles expert models to be functionally orthogonal to other tasks, using only unlabeled auxiliary data, and improves merging performance by up to 3.8% and generalization to unseen domains by up to 2.3%. The method is robust to the source of auxiliary input and less sensitive to hyperparameter tuning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 3

Model Merging via Data-Free Covariance Estimation

The paper introduces a data‑free method for model merging that estimates per‑layer covariance matrices directly from difference matrices, eliminating the need for auxiliary data. This approach reduces computational costs while maintaining a principled interference‑minimization framework. Experiments on vision and language benchmarks with models from 86 M to 7 B parameters show that the method outperforms existing data‑free merging techniques.

By Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau
benchmarks
More like this →
Hugging Face Trending Papers
Jun 25

Learning to Recover Task Experts from a Multi-Task Merged Model

Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference.

nlp
More like this →
arXiv Machine Learning
Sep 22

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.

By Jingang Zhou, Haiyang Guo, Yuan Ma, Han Zhu, Xu-Yao Zhang
llmsfine-tuning
More like this →