FiLoRA is an instruction‑conditioned, parameter‑efficient adaptation framework that allows controllable modulation of feature reliance in multimodal foundation models. It decomposes adaptation into feature‑aligned low‑rank modules and uses instruction‑conditioned gating so that natural language instructions can act as computation‑level control signals over internal representations. Experiments on classification and generative multimodal tasks show that FiLoRA consistently and interpretable shifts feature reliance, amplifying or suppressing feature groups per instruction without changing task semantics.
arXiv:2608.30712v1 Announce Type: new Abstract: Large multimodal models follow instructions about what to generate, but not necessarily about what evidence to rely on. Hence, models may continue to d...
arXiv:2506. 01850v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs).
arXiv:2608. 05000v1 Announce Type: cross Abstract: Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining.
arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.
arXiv:2605.30117v2 Announce Type: replace Abstract: Understanding how Vision-Language-Action (VLA) models transform multimodal knowledge into embodied control remains an open challenge. We present VL...
arXiv:2608. 07525v1 Announce Type: cross Abstract: Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications.
Dynamic Cross-Modal Prompt Generation (DRAPE) is a prompt-learning framework designed for Multimodal Continual Instruction Tuning (MCIT). It generates continuous, instance-specific soft prompts by cross-attending textual instructions with visual patch features, rather than selecting from a fixed prompt pool. DRAPE also incorporates null‑space gradient projection to reduce forgetting and uses CLIP‑based prototype routing for task‑label‑free inference, achieving state‑of‑the‑art results on MCIT benchmarks.
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
arXiv:2503. 06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as audio and images while effectively leveraging that knowledge remains challenging.
arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.
arXiv:2607. 03738v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context.
arXiv:2507. 19634v4 Announce Type: replace-cross Abstract: Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework.