arXiv Machine Learning
23h ago
CORAM: Coherent Orthogonal Rotation for Model Merging
arXiv:2608. 17366v1 Announce Type: new Abstract: Merging finetuned models combines specialized capabilities without joint training or access to the original data.
CORAM: Coherent Orthogonal Rotation for Model Merging
Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update.
Related stories
arXiv AI
Aug 11
Shape Mutating Expert Compression:LorExperts and BTExperts
arXiv:2608. 07814v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices.
arXiv Machine Learning
Jul 24
CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging
arXiv:2607. 20561v1 Announce Type: new Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time.
arXiv Machine Learning
Jun 3
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
arXiv:2606. 03723v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment.
arXiv Machine Learning
Jun 19
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
arXiv Machine Learning
Jun 18
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
arXiv Machine Learning
Jun 2
Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.
arXiv AI
Jul 15
Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
arXiv:2607. 11997v1 Announce Type: cross Abstract: Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training.
arXiv AI
Jul 21
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.
arXiv Machine Learning
Aug 10
Training-free Task Classification for Multi-Task Model Merging
arXiv:2606. 22589v2 Announce Type: replace Abstract: Ever since the advent of foundation models and the pre-training-finetuning paradigm, there have been numerous efforts to merge multiple task-specific experts into a single multi-task model.
arXiv Machine Learning
Jul 7
CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation
arXiv:2607. 02576v1 Announce Type: cross Abstract: Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection.
arXiv AI
Jul 21
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
arXiv:2607. 18026v1 Announce Type: new Abstract: Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment?