The paper introduces a new multi‑task model‑merging framework that tackles task interference by projecting task vectors into a high‑dimensional sparse feature space using Sparse Autoencoders, enabling feature‑level disentanglement before fusion. It also proposes a lightweight Group‑Ranked Zeroth‑Order Optimizer to identify task‑critical layers for selective merging, reducing computational overhead. Experiments on Qwen2.5‑1.5B and Qwen2.5‑7B show consistent performance gains over several baselines across reasoning, code generation, instruction following, and general knowledge tasks, with a 2.78% improvement in a highly conflicting four‑task setting.
By Yihang Zhang, Shengke Sun, Junjie Wen, Feng Zeng
arXiv:2312. 06173v2 Announce Type: replace Abstract: Merging models fine-tuned from a common, extensively pre-trained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multi-task model that performs well across diverse tasks.
By Anke Tang, Xianglin Luo, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, Dacheng Tao
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
By Ningyuan Shi, Zhipeng Zhou, Hao Wang, Chunyan Miao, Peilin Zhao
arXiv:2606. 19164v1 Announce Type: cross Abstract: Model merging aims to enable multi-task learning by integrating the capabilities of multiple models fine-tuned from the same pre-trained checkpoint into a single model.
By Longhua Li, Lei Qi, Xin Geng, Qi Tian
The paper introduces Resolving Interference (RI), a lightweight adaptation framework that reduces cross‑task interference when merging expert models. RI disentangles expert models to be functionally orthogonal to other tasks, using only unlabeled auxiliary data, and improves merging performance by up to 3.8% and generalization to unseen domains by up to 2.3%. The method is robust to the source of auxiliary input and less sensitive to hyperparameter tuning.
By Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman
CRLoMTL is a parameter‑efficient multi‑task learning framework that mitigates task interference by operating in a low‑rank parameter‑update space rather than gradient space. It learns task‑specific LoRA updates and fuses them through a shared‑subspace resolver based on singular value decomposition, producing a single deployment‑time update that can be folded into the backbone. Experiments on Swin Transformer backbones across PASCAL‑Context, NYUD‑v2, and CelebA show that CRLoMTL improves accuracy over single‑task baselines while training only a small fraction of the parameters.
By Ahmed Agiza, Tasneem Shaffee, Rahil Badkul, Sherief Reda
arXiv:2610.07990v1 Announce Type: new
Abstract: Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tas...
By Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee
CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.
By Jingang Zhou, Haiyang Guo, Yuan Ma, Han Zhu, Xu-Yao Zhang
arXiv:2606. 22589v2 Announce Type: replace Abstract: Ever since the advent of foundation models and the pre-training-finetuning paradigm, there have been numerous efforts to merge multiple task-specific experts into a single multi-task model.
By Jungyong Son, Jinwook Jung, Sungyong Baik
The paper introduces DiGA, a Disentangled Geometry-Aware framework for merging pretrained models. DiGA orthogonally decomposes each task vector into components tied to distinct geometric attributes, aggregates these components independently, and then recombines them, thereby preserving each component’s geometric identity. Experiments across various models, tasks, and merging methods show that DiGA improves merged-model performance and reduces capability degradation.
By Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Mengjie Zhao, Yunpu Ma, Kang Liu, Zihan Wang, Shi Feng, Daling Wang, Hinrich Sch\"utze
arXiv:2610.29110v1 Announce Type: cross
Abstract: Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In suc...
By Di Wu, Shuaidong Pan
arXiv:2606.19164v4 Announce Type: replace-cross
Abstract: Model merging enables training-free multi-task learning by aggregating knowledge from models fine-tuned from a shared checkpoint, but their t...
By Longhua Li, Lei Qi, Qi Tian, Xin Geng