The paper introduces a new multi‑task model‑merging framework that tackles task interference by projecting task vectors into a high‑dimensional sparse feature space using Sparse Autoencoders, enabling feature‑level disentanglement before fusion. It also proposes a lightweight Group‑Ranked Zeroth‑Order Optimizer to identify task‑critical layers for selective merging, reducing computational overhead. Experiments on Qwen2.5‑1.5B and Qwen2.5‑7B show consistent performance gains over several baselines across reasoning, code generation, instruction following, and general knowledge tasks, with a 2.78% improvement in a highly conflicting four‑task setting.
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in m...
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
arXiv:2312. 06173v2 Announce Type: replace Abstract: Merging models fine-tuned from a common, extensively pre-trained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multi-task model that performs well across diverse tasks.
arXiv:2606. 19164v1 Announce Type: cross Abstract: Model merging aims to enable multi-task learning by integrating the capabilities of multiple models fine-tuned from the same pre-trained checkpoint into a single model.
arXiv:2606. 22589v2 Announce Type: replace Abstract: Ever since the advent of foundation models and the pre-training-finetuning paradigm, there have been numerous efforts to merge multiple task-specific experts into a single multi-task model.
arXiv:2610.07990v1 Announce Type: new Abstract: Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tas...
The paper introduces Resolving Interference (RI), a lightweight adaptation framework that reduces cross‑task interference when merging expert models. RI disentangles expert models to be functionally orthogonal to other tasks, using only unlabeled auxiliary data, and improves merging performance by up to 3.8% and generalization to unseen domains by up to 2.3%. The method is robust to the source of auxiliary input and less sensitive to hyperparameter tuning.
CRLoMTL is a parameter‑efficient multi‑task learning framework that mitigates task interference by operating in a low‑rank parameter‑update space rather than gradient space. It learns task‑specific LoRA updates and fuses them through a shared‑subspace resolver based on singular value decomposition, producing a single deployment‑time update that can be folded into the backbone. Experiments on Swin Transformer backbones across PASCAL‑Context, NYUD‑v2, and CelebA show that CRLoMTL improves accuracy over single‑task baselines while training only a small fraction of the parameters.
CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.
arXiv:2610.29110v1 Announce Type: cross Abstract: Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In suc...
arXiv:2510.14697v2 Announce Type: replace Abstract: Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training. In recent m...
arXiv:2512. 01461v2 Announce Type: replace Abstract: Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training.