The paper introduces SADA-Merging, a new data‑free model merging framework that addresses limitations of existing subspace‑based methods. It allocates subspace capacity based on each task’s spectral complexity, adapts spectral preservation to task‑specific plasticity, and applies depth‑dependent anchoring to mitigate projection distortion. Experiments show that SADA‑Merging consistently outperforms current data‑free merging techniques across various task scales and adaptation settings.
By Ruxi Gu, Zilei Wang, Wei Wang
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
By Ningyuan Shi, Zhipeng Zhou, Hao Wang, Chunyan Miao, Peilin Zhao
arXiv:2606. 19164v1 Announce Type: cross Abstract: Model merging aims to enable multi-task learning by integrating the capabilities of multiple models fine-tuned from the same pre-trained checkpoint into a single model.
By Longhua Li, Lei Qi, Xin Geng, Qi Tian
arXiv:2607. 20561v1 Announce Type: new Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time.
By Keumseo Ryum, Joonhyuk Kang
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
By Yuchen Liu, Zongzhen Yang, Binhang Qi, Hailong Sun, Xiang Gao
The paper introduces Net Utility, a data‑free metric for selecting which singular directions of low‑rank adapters (LoRAs) to keep when merging across tasks. By scoring each direction for task utility and interference, and then globally selecting the highest‑scoring directions under a total budget constraint, the method avoids the uniform‑budget assumption that hampers existing merging techniques. Experiments on vision and language tasks show that Net Utility‑based rank allocation yields about a 2% performance gain over other merging methods.
By Avinash Amballa, Yashas Malur Saidutta, Wenbo Li, Lazar Valkov, Srinivas Chappidi
arXiv:2606. 22589v2 Announce Type: replace Abstract: Ever since the advent of foundation models and the pre-training-finetuning paradigm, there have been numerous efforts to merge multiple task-specific experts into a single multi-task model.
By Jungyong Son, Jinwook Jung, Sungyong Baik
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
By Lin Tang, Wei Zhang, Jing Li, Hongyu Chen, Ming Zhao, Yuxuan Wang
arXiv:2607.20561v2 Announce Type: replace-cross
Abstract: LoRA merging methods increasingly operate on the low-rank structure of task updates, yet how the common subspace is estimated and how coeffic...
By Keumseo Ryum, Joonhyuk Kang
ResMerge is a new framework for merging large language models trained via reinforcement learning. It separates each model’s task vector into a leading spectral head and a residual component, finding that both parts contain valuable behavior knowledge but behave differently during merging. The method builds a stable residual backbone using Spherical Residual Consensus Adaptation and then adds a lightweight head correction module that activates only when experts agree, leading to better preservation of expert capabilities compared to existing merging baselines.
By Yandu Sun, Zhiyan Hou, Hongyan An, Weizhen Wang, Haokai Ma, Yuheng Jia, Junfeng Fang, Haiyun Guo, Jinqiao Wang
arXiv:2512. 01461v2 Announce Type: replace Abstract: Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training.
By Kuangpu Guo, Aijing Yu, Jian Liang, Yuhe Ding, Zilei Wang, Ran He, Tieniu Tan
arXiv:2606. 03723v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment.
By Zhengbao He, Ruiqi Ding, Zhehao Huang, Ruikai Yang, Tao Li, Xiaolin Huang