The paper investigates whether model merging requires all experts to use the same LoRA rank. Experiments with 36 experts across three domains and multiple ranks show that allowing each expert its own optimal rank improves multi‑domain accuracy by up to 1.8 percentage points, while traditional uniform‑rank methods can even degrade performance. The authors attribute this to how merge methods select spectral components, arguing that rank can be chosen per expert when the merge operator does not rely on magnitude alone.
arXiv:2607. 11997v1 Announce Type: cross Abstract: Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training.
CORAM (Coherent Orthogonal Rotation for Model Merging) is a new method for combining fine‑tuned models without joint training or access to original data. It partitions each target weight matrix into row slices, represents each expert slice with its singular value decomposition in the base‑model SVD frame, and merges the task‑specific factors on their corresponding manifolds. The approach includes an amplification coefficient to counteract manifold averaging contraction, spread slicing to balance highly updated rows, and a residual pathway for non‑target layers, achieving improvements over existing orthogonal merging techniques across multiple model families and scales.
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
arXiv:2609.06072v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-w...
The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.
CORAM introduces a new approach to merging fine‑tuned models by partitioning each target weight matrix into row slices and representing each slice with its singular value decomposition in the base‑model’s SVD frame. The method performs manifold averaging of task‑specific factors and applies an amplification coefficient to counteract contraction, with the coefficient’s scale estimated from update norms and its restoration strength chosen from expert update dispersion. Across multiple model families and scales, CORAM outperforms the prior OrthoMerge technique by up to 1.35 points and matches or exceeds the strongest weight‑space baselines.
SMAT (Simple and Efficient Merge-Aware Training) is a method that jointly optimizes expert loss and expected loss at simulated merged parameters, addressing the shortcomings of standard expert training for model merging. It models common merging operations—Scale, Mask, and Perturb—by sampling scaling coefficients, masks, and additive noise, and introduces efficient techniques such as periodic scheduling, kernel fusion, and parameter storage switching to keep training overhead minimal. Across multiple language and vision-language backbones, SMAT improves mean scores by 1.07–2.16 points over the strongest baseline while adding less than 2% training-time overhead.
The paper introduces a method for predicting and preventing merge collapse when combining large language models fine‑tuned from a shared base. By measuring the variance of specialists’ task vectors—termed interference—a pre‑merge score identifies destructive merges. The authors present PRISM, an operator that averages task vectors and then soft‑thresholds each layer based on interference, which successfully preserves model performance without additional data or tuning.
arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.
arXiv:2607. 16062v1 Announce Type: cross Abstract: Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable.
arXiv:2608. 09201v1 Announce Type: new Abstract: Dense expert merging combines domain-specialized language models into one single checkpoint, typically by admitting task-vector support in weight space.
arXiv:2605.18904v2 Announce Type: replace-cross Abstract: Model merging consolidates independently fine-tuned checkpoints without joint training or access to task data. Dynamic methods retain task-sp...