FeatCal is a post‑merging calibration method that reduces feature drift between a merged model and its task experts by adjusting weights layer by layer in forward order using a small calibration set. It employs a closed‑form update that avoids gradient descent or iterative optimization. On CLIP and GLUE benchmarks, FeatCal outperforms existing calibration baselines, achieving higher accuracy with fewer calibration examples and lower computational cost.
arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.
SMAT (Simple and Efficient Merge-Aware Training) is a method that jointly optimizes expert loss and expected loss at simulated merged parameters, addressing the shortcomings of standard expert training for model merging. It models common merging operations—Scale, Mask, and Perturb—by sampling scaling coefficients, masks, and additive noise, and introduces efficient techniques such as periodic scheduling, kernel fusion, and parameter storage switching to keep training overhead minimal. Across multiple language and vision-language backbones, SMAT improves mean scores by 1.07–2.16 points over the strongest baseline while adding less than 2% training-time overhead.
ReForge is a bilevel optimization framework that refines merged models by treating module-wise refinement as Bayesian linear regression with an anchor-centered prior. The inner level produces a closed‑form MAP estimate from unlabeled calibration activations, while the outer level employs Bayesian optimization to jointly select regularization strengths and assembly scales using validation data. A data‑free variant replaces activation statistics with task‑vector Grams, enabling refinement without calibration examples, and across extensive vision and language benchmarks ReForge consistently outperforms existing plug‑and‑play anchor baselines, achieving significant accuracy gains on large‑scale tasks such as 20‑task ViT‑B/32 and eight‑task ViT‑L/14.
arXiv:2603. 12433v3 Announce Type: replace-cross Abstract: Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility.
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
arXiv:2610.07990v1 Announce Type: new Abstract: Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tas...
Merge++ is a post‑hoc refinement technique for model merging that synthesizes task‑representative images by inverting expert checkpoints and then distills expert knowledge into a merged model. It operates without any additional data beyond the checkpoints and can be applied universally across existing weight‑space merging algorithms. Experiments show consistent improvements, with average gains of +2 to +8 points and up to +25.9 on specific configurations.
UniMergeTrack introduces a hierarchical model merging strategy for RGB-X visual object tracking, combining independently trained modality-specific trackers into a unified framework. The method merges transferable backbones using high‑pass‑filtered merging, aligns tracking heads with optimal transport‑guided channel alignment, and fine‑tunes with distribution alignment to integrate cross‑task knowledge while preserving modality cues. Experiments on multiple benchmarks show state‑of‑the‑art or competitive performance across RGB‑Thermal, RGB‑Event, and RGB‑Depth tracking tasks, with strong generalization to unseen datasets.
arXiv:2607. 11997v1 Announce Type: cross Abstract: Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training.
arXiv:2607.20561v2 Announce Type: replace-cross Abstract: LoRA merging methods increasingly operate on the low-rank structure of task updates, yet how the common subspace is estimated and how coeffic...
arXiv:2608.31096v1 Announce Type: cross Abstract: Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while...