arXiv Computation and Language August 28, 2026 By Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Read the original on arXiv Computation and Language →

CRAM (Centroid‑Routing and Adaptive MoE) is a method for Multimodal Continual Instruction Tuning that isolates task‑specific patterns into independent modules to reduce catastrophic forgetting. It uses adaptive‑rank instantiation to allocate only the necessary parameters for new tasks, and centroid‑guided routing with an orthogonality penalty to reuse existing experts while preventing interference. Experiments on diverse benchmarks show CRAM outperforms existing approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Sep 30

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

arXiv:2609.37889v1 Announce Type: cross Abstract: Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks whil...

By Tao Hu, Zhinuo Zhou, Xialiang Tong, De-Chuan Zhan, Da-Wei Zhou

llms rag fine-tuning multimodal benchmarks

arXiv AI

3d ago

CART: Compatible Routing for Multimodal Continual Instruction Tuning

The paper introduces CART, a method for Multimodal Continual Instruction Tuning that coordinates adaptation between a projector and a language model by using task‑specific experts and weighted mixtures. CART jointly infers expert weights based on relevance and cross‑module compatibility, and it controls model growth through relevance‑guided expert consolidation. Experiments show that CART achieves state‑of‑the‑art performance on multimodal continual learning tasks.

By Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Sep 29

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks while preserving previously learned knowledge. Existing...

llms rag fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 27

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

The paper introduces SAME (Stabilized Mixture-of-Experts) to address challenges in Multimodal Continual Instruction Tuning (MCIT) for large language models. SAME mitigates router drift by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions, while preventing expert drift through curvature‑aware scaling that uses historical input covariance without rehearsal. The method also employs adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross‑task interference, and demonstrates state‑of‑the‑art performance on a new long‑task‑sequence benchmark.

By Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 20

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.

By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

CART: Compatible Routing for Multimodal Continual Instruction Tuning

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning