arXiv Computation and Language
Sep 1

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

The study examines how vision‑language models handle multi‑turn pragmatic interpretation in iterated reference games, where participants repeatedly identify novel referents using language. Researchers compared human performance with that of several models, manipulating context by varying its amount, order, and relevance. While humans consistently performed well, the models could use prior context but struggled to build relevant context for effective interpretation, indicating missing core skills for efficient linguistic collaboration.

By Alvin Wei Ming Tan, Ben Prystawski, Veronica Boyce
arXiv Computation and Language
Aug 25

PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understa...

By S{\l}awomir Dadas, Micha{\l} Pere{\l}kiewicz, Rafa{\l} Po\'swiata, Ma{\l}gorzata Gr\k{e}bowiec, Bart{\l}omiej Jaworski, Izabela Wo\'zniakowska
arXiv Computation and Language
Sep 3

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

MemeCULT-1K is a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, each paired with a cultural context note and three human-written explanations, plus an additional set of 54 Bengali regional dialect memes. The study evaluates thirteen vision‑language models under meme‑only and context‑aware settings, showing that providing minimal cultural context consistently improves performance across all models and languages. Error analysis indicates closed‑source models struggle with entity and reference misidentification, while open‑source models are limited by broader cultural knowledge gaps, especially in linguistic and phonological aspects.

By Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir, Mueeze Al Mushabbir, Mohammed Saidul Islam, Mir Rayat Imtiaz Hossain, Md Tahmid Rahman Laskar, Sabbir Ahmed
Hugging Face Trending Papers
Jul 8

Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts

Claims about the universality of human concepts have been predominantly assessed through linguistic similarity across languages and cultures. However, words are effective as communication devices because they compress rich experiential variation into shared conventions, potentially obscuring hidden individual and cultural differences in how concepts are mentally represented.