arXiv AI
Aug 18
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
arXiv:2608. 00410v2 Announce Type: replace Abstract: Human language is highly polysemous.
Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picture
Related stories
Hugging Face Trending Papers
Oct 22
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, crafted so that all three modalities together yield a unique answer while any single modality leaves the answer ambiguous. The benchmark contains 327 hand‑written queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans significantly in scene‑level disambiguation, especially when answer options are removed.
arXiv Computation and Language
Oct 23
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, constructed so that the full triplet yields a unique answer while any single modality removed makes the answer ambiguous. The benchmark includes 327 queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans in scene‑level disambiguation, especially when answer options are removed.
arXiv Computation and Language
Sep 1
Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation
The study examines how vision‑language models handle multi‑turn pragmatic interpretation in iterated reference games, where participants repeatedly identify novel referents using language. Researchers compared human performance with that of several models, manipulating context by varying its amount, order, and relevance. While humans consistently performed well, the models could use prior context but struggled to build relevant context for effective interpretation, indicating missing core skills for efficient linguistic collaboration.
arXiv Computation and Language
Aug 25
PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding
arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understa...
arXiv AI
Oct 26
CEMIR vs. FISER: a comparative study of reasoning-based and enrichment-based approaches to pragmatic intent recognition
The paper compares two approaches to pragmatic intent recognition in human‑robot interaction: FISER, which uses structured symbolic reasoning, and CEMIR, which enriches textual prompts with synthetic narratives and images. Experiments across four large language models on the HandMeThat v2 benchmark show no statistically significant performance difference between the two methods, regardless of model size or ambiguity difficulty. The results indicate that when task constraints are already present in the text, multimodal enrichment offers no measurable advantage over structured reasoning.
arXiv Computation and Language
Oct 7
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
AVMeme Exam is a human‑curated benchmark featuring over a thousand iconic Internet audio‑visual clips—including speech, songs, music, and sound effects—each paired with a unique Q&A that probes understanding from surface content to context, emotion, usage, and world knowledge. The benchmark also provides metadata such as original year, transcript, summary, and sensitivity. Evaluations of state‑of‑the‑art multimodal large language models (MLLMs) and human participants reveal that current models perform poorly on textless music and sound effects and struggle to think in cultural and contextual terms compared to surface content.
arXiv Computation and Language
Sep 3
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
MemeCULT-1K is a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, each paired with a cultural context note and three human-written explanations, plus an additional set of 54 Bengali regional dialect memes. The study evaluates thirteen vision‑language models under meme‑only and context‑aware settings, showing that providing minimal cultural context consistently improves performance across all models and languages. Error analysis indicates closed‑source models struggle with entity and reference misidentification, while open‑source models are limited by broader cultural knowledge gaps, especially in linguistic and phonological aspects.
arXiv AI
Oct 9
Whose Ground Truth? Embracing Ambiguity in Human-Centered AI
The paper titled "Whose Ground Truth? Embracing Ambiguity in Human-Centered AI" argues that current AI systems assume a single definitive ground truth, which ignores the inherent ambiguity in human interpretations. It calls for modeling the full space of plausible human judgments, distinguishing true ambiguity from noise, and suggests that this shift should influence how AI is represented, learned, evaluated, deployed, and governed.
arXiv AI
Oct 2
VIDA: A Dataset for Visually Dependent Ambiguity in Multimodal Machine Translation
VIDA (Visually-Dependent Ambiguity) is a new dataset comprising 2,500 curated instances designed to test whether multimodal machine translation models can use visual input to resolve ambiguous source spans. The authors also introduce Disambiguation-Centric Metrics, which employ an LLM-as-a-judge classifier to verify correct span-level resolution. Experiments with recent large vision‑language models show that visual disambiguation remains difficult, and that chain‑of‑thought supervised fine‑tuning yields stronger out‑of‑distribution performance, especially for collective‑noun ambiguities.
arXiv Computer Vision
Sep 1
TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film
arXiv:2608.30068v1 Announce Type: new Abstract: Films communicate through deliberate creative choices, including lighting, color, composition, editing, dialogue, music, and sound. Humans naturally in...
Hugging Face Trending Papers
Jul 8
Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts
Claims about the universality of human concepts have been predominantly assessed through linguistic similarity across languages and cultures. However, words are effective as communication devices because they compress rich experiential variation into shared conventions, potentially obscuring hidden individual and cultural differences in how concepts are mentally represented.