Hyper-LLaVA introduces a hyperbolic uncertainty‑aware, modality‑balanced routing mechanism for multimodal continual instruction tuning. It improves intra‑modality task matching by evaluating sample‑to‑task distribution similarity in hyperbolic space and mitigates unreliable modality influence by quantifying task‑matching ambiguity to adaptively balance across modalities. The approach achieves significant performance gains over existing methods.
arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.
The paper introduces SAME (Stabilized Mixture-of-Experts) to address challenges in Multimodal Continual Instruction Tuning (MCIT) for large language models. SAME mitigates router drift by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions, while preventing expert drift through curvature‑aware scaling that uses historical input covariance without rehearsal. The method also employs adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross‑task interference, and demonstrates state‑of‑the‑art performance on a new long‑task‑sequence benchmark.
arXiv:2607. 05438v1 Announce Type: cross Abstract: Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images.
The paper introduces Hyperbolic Multimodal Continual Learning (HMCL), a method that preserves the Lorentz geometry of hyperbolic multimodal models during sequential updates. By restricting all modalities to a shared hyperbolic isometry, HMCL formulates a joint closest‑admissible (CA) correction—along with a minimal‑rotation (MR) variant—to adjust AdamW updates while maintaining task performance. Experiments on a 16‑task classification‑retrieval stream with three hyperbolic backbones show that HMCL-CA achieves the highest overall score, reduces geometric drift by up to 95.5 %, and improves semantic hierarchy preservation on ImageNet‑WordNet. whyItMatters":"The study demonstrates that explicitly maintaining hyperbolic geometry during continual learning yields superior performance and reduced representation drift compared to existing baselines."
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
CRAM (Centroid‑Routing and Adaptive MoE) is a method for Multimodal Continual Instruction Tuning that isolates task‑specific patterns into independent modules to reduce catastrophic forgetting. It uses adaptive‑rank instantiation to allocate only the necessary parameters for new tasks, and centroid‑guided routing with an orthogonality penalty to reuse existing experts while preventing interference. Experiments on diverse benchmarks show CRAM outperforms existing approaches.
arXiv:2608. 13911v1 Announce Type: new Abstract: Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may contain different partial modality subsets.
arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.
arXiv:2608. 15516v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation.
arXiv:2607. 27260v1 Announce Type: new Abstract: Multimodal continual learning (MMCL) aims to learn emerging knowledge from multimodal data while preserving knowledge.
arXiv:2502. 04411v3 Announce Type: replace-cross Abstract: Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one.
The paper examines how skill representations influence selection in a multimodal video agent harness called Tinycloud. It compares two types of skill representations—tool-skills and workflow-skills—and two prompt surfaces—full inlined bodies and one-line listings—across three exposure regimes. The study finds that full autoload exposure consistently selects the correct skill, while partial exposure can cause lexical competition that misroutes tasks, highlighting that in-prompt exposure is not always beneficial.