The article surveys model fusion for large language models, defining it as the integration of capabilities from source models into a single target model. It categorizes prior work into three levels—parameter-level, representation-level, and behavior-level fusion—and reviews related metrics, benchmarks, and applications. The survey also outlines current challenges and future research directions, providing a comprehensive map of the field.
arXiv:2507. 10540v3 Announce Type: replace Abstract: The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks.
FusionEval is a new benchmark for evaluating model fusion techniques that combine multiple expert models into a single large language model. It provides controlled expert pools trained from Qwen3 and Llama on domains such as math, code, science, instruction following, and tool use, and evaluates capability retention, computational cost, runtime, and GPU memory across eight public benchmarks. Experiments show that training‑free fusion methods keep reasoning performance but hurt tool use, while MOPD maintains tool use and balances capabilities at the cost of extra data preparation and training.
arXiv:2607. 18026v1 Announce Type: new Abstract: Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment?
InfiFPO is a new preference optimization technique for implicit model fusion that replaces the reference model in Direct Preference Optimization with a fused source model, preserving probability information and avoiding complex vocabulary alignment. By incorporating probability clipping and max‑margin fusion, it aligns a pivot model with human preferences while distilling knowledge from multiple source models. Experiments on 11 benchmarks show that InfiFPO consistently outperforms existing fusion and preference optimization methods, raising Phi‑4’s average score from 79.95 to 83.33 and improving performance in mathematics, coding, and reasoning tasks.
arXiv:2607. 03461v1 Announce Type: cross Abstract: World models aim to capture environment dynamics in ways that support perception, reasoning, and action, and have recently become a central direction in Vision-Language-Action-World (VLAW) modeling.
arXiv:2603.13966v3 Announce Type: replace Abstract: Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pi...
arXiv:2610.19825v1 Announce Type: cross Abstract: Integrating world models into Vision-Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynam...
arXiv:2601. 03100v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) typically rely on a single late-layer feature from a frozen vision encoder, leaving the encoder's rich hierarchy of visual cues under-utilized.
The paper introduces Uni-SafeBench, a safety benchmark designed to evaluate the holistic safety of Unified Multimodal Large Models (UMLMs) across six safety categories and seven task types. It also presents Uni-Judger, a framework that separates contextual safety from intrinsic safety to enable rigorous assessment. Evaluations reveal that current unified models do not consistently maintain the safety alignment of their underlying language models, and open‑source UMLMs perform significantly worse on safety than specialized multimodal models, especially in generation tasks.
CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.
arXiv:2607. 08974v1 Announce Type: cross Abstract: Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control.
arXiv:2608.22370v1 Announce Type: new Abstract: Task-specific LoRA adapters offer a modular way to specialize large language and vision-language models. However, existing adapter composition methods...