arXiv:2607. 11997v1 Announce Type: cross Abstract: Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training.
By Nikita Kozodoi, Zainab Afolabi, Jack Butler
CORAM (Coherent Orthogonal Rotation for Model Merging) is a new method for combining fine‑tuned models without joint training or access to original data. It partitions each target weight matrix into row slices, represents each expert slice with its singular value decomposition in the base‑model SVD frame, and merges the task‑specific factors on their corresponding manifolds. The approach includes an amplification coefficient to counteract manifold averaging contraction, spread slicing to balance highly updated rows, and a residual pathway for non‑target layers, achieving improvements over existing orthogonal merging techniques across multiple model families and scales.
By Xinyi Sui, Ziran Liu, Nam Ling, Wei Wang, Wei Jiang
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
By Lin Tang, Wei Zhang, Jing Li, Hongyu Chen, Ming Zhao, Yuxuan Wang
arXiv:2609.06072v1 Announce Type: cross
Abstract: Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-w...
By Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong
The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.
By Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
CORAM introduces a new approach to merging fine‑tuned models by partitioning each target weight matrix into row slices and representing each slice with its singular value decomposition in the base‑model’s SVD frame. The method performs manifold averaging of task‑specific factors and applies an amplification coefficient to counteract contraction, with the coefficient’s scale estimated from update norms and its restoration strength chosen from expert update dispersion. Across multiple model families and scales, CORAM outperforms the prior OrthoMerge technique by up to 1.35 points and matches or exceeds the strongest weight‑space baselines.