arXiv AI By Huyu Wu, Meng Tang, Xinhan Zheng, Haiyun Jiang

When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models

Read the original on arXiv AI →

The paper investigates the problem of text dominance in multimodal large language models (MLLMs), where models rely heavily on textual input while underusing other modalities such as images, videos, audio, time‑series, and graphs. It introduces two metrics, the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI), to quantify this imbalance and finds that text dominance is significant across all tested modalities. The authors identify three causes—attention dilution from token redundancy, fusion architecture design, and task formulations favoring text—and propose a simple token compression technique that successfully reduces MDI in LLaVA‑7B from 10.23 to 0.86, offering a path toward more balanced multimodal models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
Oct 12

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.

By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
arXiv Machine Learning
Aug 28

Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

The paper introduces Inverted Asymmetric Fusion (IAF) to address strong-modality collapse in multimodal learning, where dominant modalities are degraded during fusion. IAF preserves the dominant modality by passing it unchanged and letting weaker modalities attend to it, while also strengthening weaker modalities via Modality-Aware Knowledge Distillation. Experiments on MultiHuSE, UR-FUNNY, and MUStARD show that IAF maintains unimodal performance and improves over the best unimodal baseline by up to 8.25%.

By Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat
arXiv Machine Learning
Jun 2

Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality

arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
arXiv AI
3d ago

DRAMA: Dissecting Attention Redundancy for Accelerating Multimodal Diffusion Large Language Models

DRAMA is a training‑free acceleration framework for multimodal diffusion large language models that dissects attention redundancy by probing attention tensors along spatial and temporal axes. It identifies two key redundancy patterns: a sparse subset of visual tokens consistently receives most attention, and attention distributions stabilize quickly after an initial phase. Leveraging these insights, DRAMA introduces Spatial‑Redundancy‑Aware Visual Token Pruning and Sufficiency‑Guided Adaptive Stopping, achieving an average 18.67× inference speedup across six vision‑language benchmarks while preserving competitive accuracy.

By Yi-Hsin Hung, Fangfu Liu, Shao-Yuan Lo, Chu-Song Chen
arXiv Machine Learning
Jun 3

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

arXiv:2603. 01471v3 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.

By Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi