arXiv Machine Learning
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computer Vision
Sep 22
Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging
The paper introduces SADA-Merging, a new data‑free model merging framework that addresses limitations of existing subspace‑based methods. It allocates subspace capacity based on each task’s spectral complexity, adapts spectral preservation to task‑specific plasticity, and applies depth‑dependent anchoring to mitigate projection distortion. Experiments show that SADA‑Merging consistently outperforms current data‑free merging techniques across various task scales and adaptation settings.
arXiv Machine Learning
Jun 18
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
arXiv AI
Jun 18
Essential Subspace Merging for Multi-Task Learning
arXiv:2606. 19164v1 Announce Type: cross Abstract: Model merging aims to enable multi-task learning by integrating the capabilities of multiple models fine-tuned from the same pre-trained checkpoint into a single model.
arXiv Machine Learning
Jul 24
CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging
arXiv:2607. 20561v1 Announce Type: new Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time.
arXiv AI
Aug 14
CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
arXiv Machine Learning
Sep 22
Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks
The paper introduces Net Utility, a data‑free metric for selecting which singular directions of low‑rank adapters (LoRAs) to keep when merging across tasks. By scoring each direction for task utility and interference, and then globally selecting the highest‑scoring directions under a total budget constraint, the method avoids the uniform‑budget assumption that hampers existing merging techniques. Experiments on vision and language tasks show that Net Utility‑based rank allocation yields about a 2% performance gain over other merging methods.