arXiv AI October 21, 2026 By Tao Hu, Da-Wei Zhou

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

Dynamic Cross-Modal Prompt Generation (DRAPE) is a prompt-learning framework designed for Multimodal Continual Instruction Tuning (MCIT). It generates continuous, instance-specific soft prompts by cross-attending textual instructions with visual patch features, rather than selecting from a fixed prompt pool. DRAPE also incorporates null‑space gradient projection to reduce forgetting and uses CLIP‑based prototype routing for task‑label‑free inference, achieving state‑of‑the‑art results on MCIT benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Sep 30

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

arXiv:2609.37889v1 Announce Type: cross Abstract: Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks whil...

By Tao Hu, Zhinuo Zhou, Xialiang Tong, De-Chuan Zhan, Da-Wei Zhou

llms rag fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Sep 29

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks while preserving previously learned knowledge. Existing...

llms rag fine-tuning multimodal benchmarks

arXiv AI

Oct 16

CART: Compatible Routing for Multimodal Continual Instruction Tuning

The paper introduces CART, a method for Multimodal Continual Instruction Tuning that coordinates adaptation between a projector and a language model by using task‑specific experts and weighted mixtures. CART jointly infers expert weights based on relevance and cross‑module compatibility, and it controls model growth through relevance‑guided expert consolidation. Experiments show that CART achieves state‑of‑the‑art performance on multimodal continual learning tasks.

By Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Aug 28

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

CRAM (Centroid‑Routing and Adaptive MoE) is a method for Multimodal Continual Instruction Tuning that isolates task‑specific patterns into independent modules to reduce catastrophic forgetting. It uses adaptive‑rank instantiation to allocate only the necessary parameters for new tasks, and centroid‑guided routing with an orthogonality penalty to reuse existing experts while preventing interference. Experiments on diverse benchmarks show CRAM outperforms existing approaches.

By Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 26

GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning

arXiv:2608.23782v1 Announce Type: new Abstract: Continual learning faces the persistent challenge of catastrophic forgetting, where sequential task updates degrade previously acquired knowledge. Whil...

By Trung-Anh Dang, Duy-Cuong Bui, Ngoc-Son Vu, Christel Vrain, Vincent Nguyen

llms efficiency benchmarks

One email a morning, machine-written

Related stories

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

CART: Compatible Routing for Multimodal Continual Instruction Tuning

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

GAP-Prompt: Gated Adaptive Prompting for Efficient Continual Learning