Merge++ is a post‑hoc refinement technique for model merging that synthesizes task‑representative images by inverting expert checkpoints and then distills expert knowledge into a merged model. It operates without any additional data beyond the checkpoints and can be applied universally across existing weight‑space merging algorithms. Experiments show consistent improvements, with average gains of +2 to +8 points and up to +25.9 on specific configurations.
By Aditya Pola, Vineeth N. Balasubramanian
arXiv:2608. 09201v1 Announce Type: new Abstract: Dense expert merging combines domain-specialized language models into one single checkpoint, typically by admitting task-vector support in weight space.
By Lingching Tung, Chi-Jui Kim, Beicheng Xu, Yuchen Wang, Bin Cui
FeatCal is a post‑merging calibration method that reduces feature drift between a merged model and its task experts by adjusting weights layer by layer in forward order using a small calibration set. It employs a closed‑form update that avoids gradient descent or iterative optimization. On CLIP and GLUE benchmarks, FeatCal outperforms existing calibration baselines, achieving higher accuracy with fewer calibration examples and lower computational cost.
By Yanggan Gu, Shuo Cai, Zihao Wang, Wenjun Wang, Yuanyi Wang, Pengkai Wang, Sirui Huang, Su Lu, Jianmin Wu, Hongxia Yang
arXiv:2609.24517v1 Announce Type: cross
Abstract: Model merging aims to combine multiple fine-tuned models derived from a common pretrained model into a single multi-task model without additional joi...
By Hyunjoong Cho, Jinhyeok Jang
arXiv:2502. 04411v3 Announce Type: replace-cross Abstract: Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one.
By Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Huacan Wang, Li Shen, Bo Li, Xiaowen Chu
arXiv:2607. 20561v1 Announce Type: new Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time.
By Keumseo Ryum, Joonhyuk Kang
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
By Yuchen Liu, Zongzhen Yang, Binhang Qi, Hailong Sun, Xiang Gao
arXiv:2608. 05250v1 Announce Type: new Abstract: Multi-task supervised fine-tuning (SFT) often casts a heterogeneous data mixture as a single optimization problem, even though different tasks may reach their best generalization at different times.
By Yue Han, Ziniu Liu
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
By Lin Tang, Wei Zhang, Jing Li, Hongyu Chen, Ming Zhao, Yuxuan Wang
arXiv:2606. 22589v2 Announce Type: replace Abstract: Ever since the advent of foundation models and the pre-training-finetuning paradigm, there have been numerous efforts to merge multiple task-specific experts into a single multi-task model.
By Jungyong Son, Jinwook Jung, Sungyong Baik
The paper investigates whether model merging requires all experts to use the same LoRA rank. Experiments with 36 experts across three domains and multiple ranks show that allowing each expert its own optimal rank improves multi‑domain accuracy by up to 1.8 percentage points, while traditional uniform‑rank methods can even degrade performance. The authors attribute this to how merge methods select spectral components, arguing that rank can be chosen per expert when the merge operator does not rely on magnitude alone.
By Zirong Xie, Zainab Afolabi, Nikita Kozodoi, Jack Butler
arXiv:2606. 03723v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment.
By Zhengbao He, Ruiqi Ding, Zhehao Huang, Ruikai Yang, Tao Li, Xiaolin Huang