arXiv AI
Aug 18
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
arXiv:2608. 00410v2 Announce Type: replace Abstract: Human language is highly polysemous.
Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
Related stories
arXiv Computation and Language
Sep 1
Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
The study examines how vision‑language models handle multi‑turn pragmatic interpretation in iterated reference games, where participants repeatedly identify novel referents using language. Researchers compared human performance with that of several models, manipulating context by varying its amount, order, and relevance. While humans consistently performed well, the models could use prior context but struggled to build relevant context for effective interpretation, indicating missing core skills for efficient linguistic collaboration.
arXiv Computation and Language
Aug 25
PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding
arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understa...
arXiv Computation and Language
Sep 3
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
MemeCULT-1K is a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, each paired with a cultural context note and three human-written explanations, plus an additional set of 54 Bengali regional dialect memes. The study evaluates thirteen vision‑language models under meme‑only and context‑aware settings, showing that providing minimal cultural context consistently improves performance across all models and languages. Error analysis indicates closed‑source models struggle with entity and reference misidentification, while open‑source models are limited by broader cultural knowledge gaps, especially in linguistic and phonological aspects.
arXiv Computer Vision
Sep 1
TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film
arXiv:2608.30068v1 Announce Type: new Abstract: Films communicate through deliberate creative choices, including lighting, color, composition, editing, dialogue, music, and sound. Humans naturally in...
Hugging Face Trending Papers
Jul 8
Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts
Claims about the universality of human concepts have been predominantly assessed through linguistic similarity across languages and cultures. However, words are effective as communication devices because they compress rich experiential variation into shared conventions, potentially obscuring hidden individual and cultural differences in how concepts are mentally represented.
arXiv Computation and Language
Sep 11
Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding
The paper introduces IRS (Incongruity-Resolution Supervision), a framework that breaks humor understanding into three parts: identifying mismatches in a visual scene, creating coherent reinterpretations of those mismatches, and aligning these interpretations with human preferences. IRS uses structured reasoning traces to guide models from visual perception to humorous interpretation, and it is evaluated on the New Yorker Cartoon Caption Contest. Experiments on 7B, 32B, and 72B models show that IRS improves caption matching and ranking, with the 72B model achieving 76.10% ranking accuracy—outperforming non-expert humans and all other multimodal baselines—and demonstrates transferable reasoning patterns in zero‑shot settings.
arXiv AI
Sep 10
NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures
NormViz introduces a new benchmark, NormViz‑Bench, comprising 3,268 contrastive image pairs from 16 countries that test AI’s ability to recognize culturally relevant visual norms. Each pair differs only in a behavior that changes its cultural interpretation, and images are labeled as conforming, violating, or irrelevant to local norms, requiring both images to be correctly classified. The benchmark shows current VLMs perform poorly, and a complementary training set, NormViz‑Train, offers a path to improve performance by teaching models to link visual perception with cultural significance.
arXiv Computation and Language
Aug 28
Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
The paper introduces a new benchmark called Speech-Augmented Visually Grounded Contrastive Triplet Benchmark, comprising 10,150 images from 18 MENA countries, each paired with a supported statement and two plausible but unsupported alternatives. It defines contrastive instability as the rate at which multimodal models fail to resolve all statements within a triplet, distinguishing fragmented reasoning from complete failure. Experiments on recent multimodal models show that shifts in modality (text vs. speech) and language (English vs. Arabic) lead to significant triplet-level inconsistencies, especially when speech is used, which are not fully reflected by overall accuracy metrics.
arXiv AI
Jun 26
What We are Missing in Multimodal LLM Evaluation?
arXiv:2606. 26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.
arXiv AI
Jul 22
Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
arXiv:2607. 19011v1 Announce Type: cross Abstract: Multimodal humor in memes, cartoons, and comics remains difficult for AI systems because intended meaning depends on non-literal mechanisms, shared cultural knowledge, and communicative intent rather than literal scene description.
arXiv Computation and Language
3d ago
Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition
arXiv:2609.07474v2 Announce Type: replace Abstract: Language models compute over tokens: language is their input, their output, and increasingly their internal representation. Whether language should...