arXiv Computation and Language August 28, 2026 By Basel Mousi, Fahim Dalvi, Shammur Chowdhury, Firoj Alam, Nadir Durrani

Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models

Read the original on arXiv Computation and Language →

The paper introduces a new benchmark called Speech-Augmented Visually Grounded Contrastive Triplet Benchmark, comprising 10,150 images from 18 MENA countries, each paired with a supported statement and two plausible but unsupported alternatives. It defines contrastive instability as the rate at which multimodal models fail to resolve all statements within a triplet, distinguishing fragmented reasoning from complete failure. Experiments on recent multimodal models show that shifts in modality (text vs. speech) and language (English vs. Arabic) lead to significant triplet-level inconsistencies, especially when speech is used, which are not fully reflected by overall accuracy metrics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jun 26

What We are Missing in Multimodal LLM Evaluation?

arXiv:2606. 26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.

By Po-han Li, Shenghui Chen, Sandeep Chinchali, Ufuk Topcu

llms multimodal benchmarks

arXiv AI

Aug 7

C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

arXiv:2608. 05381v1 Announce Type: new Abstract: Current Multimodal Large Language Models (MLLMs) can process diverse sensory inputs, yet their reasoning remains heavily biased toward a dominant modality, resulting in brittle cross-modal reasoning.

By Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu, Ponnurangam Kumaraguru

llms multimodal benchmarks

Hugging Face Trending Papers

4d ago

Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts

Indirect speech acts (ISAs) require pragmatic reasoning over context, as directive intent can- not be inferred from surface form alone. Prior text-based studies and existing multimodal benchmarks larg...

nlp multimodal benchmarks

arXiv Computation and Language

3d ago

Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts

arXiv:2608.30270v1 Announce Type: new Abstract: Indirect speech acts (ISAs) require pragmatic reasoning over context, as directive intent can- not be inferred from surface form alone. Prior text-base...

By Jaehee Kim, Ji Hoon Chung, Seoyoon Park, Unsol Kim, Kyungwon Park, Ji Hak Kim, Yi-Jun Chen, Hansaem Kim

nlp multimodal benchmarks

arXiv Computation and Language

2d ago

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

arXiv:2609.00550v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly provided with contextual evidence in heterogeneous forms: as a text passage, as a rendered im...

By Jungyeon Lee, Yejin Yoon, Taeuk Kim

llms rag fine-tuning multimodal safety

arXiv AI

Aug 26

EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$

EXAM$^2$ is a new benchmark for audio understanding that covers six languages and multiple modalities—speech, sound, music, mixed-audio, and visual images—providing 5,667 multiple-choice questions, 22,614 image instances, and 135,684 multilingual translations. It evaluates large audio language models (LALMs) and multimodal large language models (LLMs), revealing significant gaps in multilingual and cross‑modal performance. The authors also introduce Gemma3n-EXAM$^2$, a lightweight fusion model that improves multilingual results by up to 12.4% and multimodal results by 21.7% over a strong baseline.

By Jiawen Wang, Xiaoxue Gao, Zi Haur Pang, Nancy F. Chen

llms fine-tuning multimodal benchmarks

One email a morning, machine-written

Related stories

What We are Missing in Multimodal LLM Evaluation?

C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts

Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts

Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict

EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$