Early Intervention for VFM-based Multimodal Medical Image Classification

The paper introduces an Early Intervention (EI) framework for multimodal medical image classification that addresses two key challenges: limited exploitation of complementary multimodal information and scarcity of labeled data for Vision Foundation Models (VFMs). EI treats one modality as the target and uses high‑level semantic tokens from other modalities as intervention tokens to guide the target’s embedding early in the process. The authors also propose Mixture of varied‑rank LoRAs (MoR) for efficient VFM adaptation, and demonstrate the method’s effectiveness on retinal, skin, and knee medical image datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• rag
• computer-vision
• fine-tuning
• multimodal