arXiv:2607. 20561v1 Announce Type: new Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time.
arXiv:2609.24517v1 Announce Type: cross Abstract: Model merging aims to combine multiple fine-tuned models derived from a common pretrained model into a single multi-task model without additional joi...
arXiv:2606. 03723v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment.
The paper introduces SADA-Merging, a new data‑free model merging framework that addresses limitations of existing subspace‑based methods. It allocates subspace capacity based on each task’s spectral complexity, adapts spectral preservation to task‑specific plasticity, and applies depth‑dependent anchoring to mitigate projection distortion. Experiments show that SADA‑Merging consistently outperforms current data‑free merging techniques across various task scales and adaptation settings.
The paper introduces Net Utility, a data‑free metric for selecting which singular directions of low‑rank adapters (LoRAs) to keep when merging across tasks. By scoring each direction for task utility and interference, and then globally selecting the highest‑scoring directions under a total budget constraint, the method avoids the uniform‑budget assumption that hampers existing merging techniques. Experiments on vision and language tasks show that Net Utility‑based rank allocation yields about a 2% performance gain over other merging methods.
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
The paper introduces DiGA, a Disentangled Geometry-Aware framework for merging pretrained models. DiGA orthogonally decomposes each task vector into components tied to distinct geometric attributes, aggregates these components independently, and then recombines them, thereby preserving each component’s geometric identity. Experiments across various models, tasks, and merging methods show that DiGA improves merged-model performance and reduces capability degradation.
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
CORAM introduces a new approach to merging fine‑tuned models by partitioning each target weight matrix into row slices and representing each slice with its singular value decomposition in the base‑model’s SVD frame. The method performs manifold averaging of task‑specific factors and applies an amplification coefficient to counteract contraction, with the coefficient’s scale estimated from update norms and its restoration strength chosen from expert update dispersion. Across multiple model families and scales, CORAM outperforms the prior OrthoMerge technique by up to 1.35 points and matches or exceeds the strongest weight‑space baselines.
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
CORAM (Coherent Orthogonal Rotation for Model Merging) is a new method for combining fine‑tuned models without joint training or access to original data. It partitions each target weight matrix into row slices, represents each expert slice with its singular value decomposition in the base‑model SVD frame, and merges the task‑specific factors on their corresponding manifolds. The approach includes an amplification coefficient to counteract manifold averaging contraction, spread slicing to balance highly updated rows, and a residual pathway for non‑target layers, achieving improvements over existing orthogonal merging techniques across multiple model families and scales.
arXiv:2606. 00511v1 Announce Type: new Abstract: Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency.