arXiv:2608.30475v1 Announce Type: cross Abstract: We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation. It includes two tasks: (i) AynVQA, cove...
arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understa...
arXiv:2606. 05531v1 Announce Type: cross Abstract: Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and chart meaningful progress toward human-like multimodal intelligence.
BanglaVerse is a new benchmark that evaluates multilingual vision‑language models on Bengali culture, covering nine visual domains and expanding to four languages and five Bangla dialects for a total of about 32,200 artifacts. It includes visual question answering and captioning tasks built from 1,152 manually curated images. Experiments show that models perform worse on dialectal variants and that missing cultural knowledge, rather than visual grounding, is the main bottleneck.
arXiv:2607. 15740v1 Announce Type: cross Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI.
The paper introduces a new benchmark called Speech-Augmented Visually Grounded Contrastive Triplet Benchmark, comprising 10,150 images from 18 MENA countries, each paired with a supported statement and two plausible but unsupported alternatives. It defines contrastive instability as the rate at which multimodal models fail to resolve all statements within a triplet, distinguishing fragmented reasoning from complete failure. Experiments on recent multimodal models show that shifts in modality (text vs. speech) and language (English vs. Arabic) lead to significant triplet-level inconsistencies, especially when speech is used, which are not fully reflected by overall accuracy metrics.
arXiv:2608. 11907v2 Announce Type: replace-cross Abstract: As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge.
arXiv:2508. 05502v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) perform strongly in high-resource languages, yet often produce fluent but culturally "thin" descriptions in low-resource settings.
As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs).
arXiv:2609.37287v1 Announce Type: cross Abstract: Image translation is a fundamental capability of multimodal models for multilingual applications, requiring visual understanding and meaning preserva...
arXiv:2608. 11907v1 Announce Type: cross Abstract: As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge.
AVMeme Exam is a human‑curated benchmark featuring over a thousand iconic Internet audio‑visual clips—including speech, songs, music, and sound effects—each paired with a unique Q&A that probes understanding from surface content to context, emotion, usage, and world knowledge. The benchmark also provides metadata such as original year, transcript, summary, and sensitivity. Evaluations of state‑of‑the‑art multimodal large language models (MLLMs) and human participants reveal that current models perform poorly on textless music and sound effects and struggle to think in cultural and contextual terms compared to surface content.