arXiv AI

When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models

The paper investigates the problem of text dominance in multimodal large language models (MLLMs), where models rely heavily on textual input while underusing other modalities such as images, videos, audio, time‑series, and graphs. It introduces two metrics, the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI), to quantify this imbalance and finds that text dominance is significant across all tested modalities. The authors identify three causes—attention dilution from token redundancy, fusion architecture design, and task formulations favoring text—and propose a simple token compression technique that successfully reduces MDI in LLaVA‑7B from 10.23 to 0.86, offering a path toward more balanced multimodal models.

arXiv Computation and Language
Oct 12

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.

By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
arXiv Machine Learning
Aug 28

Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

The paper introduces Inverted Asymmetric Fusion (IAF) to address strong-modality collapse in multimodal learning, where dominant modalities are degraded during fusion. IAF preserves the dominant modality by passing it unchanged and letting weaker modalities attend to it, while also strengthening weaker modalities via Modality-Aware Knowledge Distillation. Experiments on MultiHuSE, UR-FUNNY, and MUStARD show that IAF maintains unimodal performance and improves over the best unimodal baseline by up to 8.25%.

By Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat
arXiv Machine Learning
Jun 2

Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality

arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
arXiv AI
3d ago

DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.

By Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song Chen
arXiv Machine Learning
Jun 3

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

arXiv:2603. 01471v3 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
arXiv Computation and Language
Oct 26

A Broader Look into MLLM's Retrieval Heads

The paper investigates how retrieval heads—sparse attention heads that transfer information from context to output—behave when multimodal large language models (MLLMs) are extended to include audio-visual inputs. Across 18 language, vision‑language, and omni‑modal checkpoints, the authors find that retrieval heads remain sparse and largely shared across modalities, with less than 2 % of heads being common to all modalities; ablating these shared heads degrades tasks such as document understanding, spatial perception, and visual mathematics. By training retrieval heads with a ranking objective, they demonstrate a 6.03‑point improvement in Recall@1 on the MMDocIR benchmark for Qwen3‑VL‑8B and improved document retrieval across three model families.

By Jia Wang, Yuqing Li, Jiangnan Li, Zheng Lin, Mo Yu
arXiv Computation and Language
Oct 28

DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

DocXEval is the first comprehensive framework that adapts existing explainable AI methods and evaluation metrics to multimodal document understanding, covering text, layout, and image modalities. The authors systematically evaluate 13 XAI methods across 3 tasks, 8 datasets, 4 models, and 17 metrics, finding that Occlusion consistently performs best in faithfulness and interpretability. The study also provides meta‑evaluation insights, diagnostic analyses of modality contributions, spatial attribution patterns, and potential dataset biases, offering a benchmark and practical guide for future research.

By Saifullah Saifullah, David Dembinsky, Stefan Agne, Andreas Dengel, Sheraz Ahmed
arXiv Computation and Language
Aug 28

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models introduces the VIG‑Sampler, a method that prioritizes tokens for decoding based on their attention to image tokens and penalizes redundancy in image‑attention distributions. The approach aims to improve the quality of multimodal generation by selecting more informative tokens during diffusion decoding. Experiments on seven captioning and VQA benchmarks with three open‑source dMLLMs show that VIG‑Sampler outperforms the Info‑Gain Sampler by an average of 19.3 CIDEr points and achieves better COCO Caption results using only half as many decoding steps.

By Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim