CORAM (Coherent Orthogonal Rotation for Model Merging) is a new method for combining fine‑tuned models without joint training or access to original data. It partitions each target weight matrix into row slices, represents each expert slice with its singular value decomposition in the base‑model SVD frame, and merges the task‑specific factors on their corresponding manifolds. The approach includes an amplification coefficient to counteract manifold averaging contraction, spread slicing to balance highly updated rows, and a residual pathway for non‑target layers, achieving improvements over existing orthogonal merging techniques across multiple model families and scales.
By Xinyi Sui, Ziran Liu, Nam Ling, Wei Wang, Wei Jiang
CORAM introduces a new approach to merging fine‑tuned models by partitioning each target weight matrix into row slices and representing each slice with its singular value decomposition in the base‑model’s SVD frame. The method performs manifold averaging of task‑specific factors and applies an amplification coefficient to counteract contraction, with the coefficient’s scale estimated from update norms and its restoration strength chosen from expert update dispersion. Across multiple model families and scales, CORAM outperforms the prior OrthoMerge technique by up to 1.35 points and matches or exceeds the strongest weight‑space baselines.
arXiv:2610.29110v1 Announce Type: cross
Abstract: Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In suc...
By Di Wu, Shuaidong Pan
arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.
By Yupeng Chang, Yuan Wu, Yi Chang
ResMerge is a new framework for merging large language models trained via reinforcement learning. It separates each model’s task vector into a leading spectral head and a residual component, finding that both parts contain valuable behavior knowledge but behave differently during merging. The method builds a stable residual backbone using Spherical Residual Consensus Adaptation and then adds a lightweight head correction module that activates only when experts agree, leading to better preservation of expert capabilities compared to existing merging baselines.
By Yandu Sun, Zhiyan Hou, Hongyan An, Weizhen Wang, Haokai Ma, Yuheng Jia, Junfeng Fang, Haiyun Guo, Jinqiao Wang
The paper introduces DiGA, a Disentangled Geometry-Aware framework for merging pretrained models. DiGA orthogonally decomposes each task vector into components tied to distinct geometric attributes, aggregates these components independently, and then recombines them, thereby preserving each component’s geometric identity. Experiments across various models, tasks, and merging methods show that DiGA improves merged-model performance and reduces capability degradation.
By Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Mengjie Zhao, Yunpu Ma, Kang Liu, Zihan Wang, Shi Feng, Daling Wang, Hinrich Sch\"utze
arXiv:2606. 11262v1 Announce Type: cross Abstract: Access control in large language models (LLMs) requires modular mechanisms to enable domain-specific behavior without retraining or cross-domain interference.
By Gowtham Sivaramakrishnan, Sarvesha Kumar Kombaiah Seetha, Kishan Gupta Balaji, Santhosh Baradwaj Vaduvur Ranganathan
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
By Ningyuan Shi, Zhipeng Zhou, Hao Wang, Chunyan Miao, Peilin Zhao
arXiv:2607. 18026v1 Announce Type: new Abstract: Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment?
By Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang, Hong Xie, Defu Lian
arXiv:2610.08312v1 Announce Type: new
Abstract: Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, rec...
By Maoqi Liu, Quan Fang, Yufei He
The paper introduces Resolving Interference (RI), a lightweight adaptation framework that reduces cross‑task interference when merging expert models. RI disentangles expert models to be functionally orthogonal to other tasks, using only unlabeled auxiliary data, and improves merging performance by up to 3.8% and generalization to unseen domains by up to 2.3%. The method is robust to the source of auxiliary input and less sensitive to hyperparameter tuning.
By Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen, Judy Hoffman
The paper investigates whether different parameter‑efficient fine‑tuning (PEFT) methods truly differ in their approach and impact. By comparing six methods on language and diffusion models, the authors find that many LoRA‑family techniques approximately preserve pretrained geometry, and that restoring slightly drifted singular‑value spectra largely maintains task performance, challenging the need for explicit geometric preservation. They also observe distinct adaptation–retention trade‑offs: LoRA limits forgetting best, DoRA scores higher on average, and PiSSA incurs more retention costs, with performance gains linked to modifications in specific spectral components.
By Yikuan Li, Pinyan Lu, Fanghui Liu