CulturalRiddles is a new benchmark consisting of original riddles created by a community effort, covering 62 cultures across 45 countries and 51 languages. The riddles are deeply rooted in their respective cultures and require abductive, compositional reasoning, factual and social knowledge, language skills, and an understanding of culturally specific associations. The authors evaluated 28 large language model configurations, finding that even state‑of‑the‑art models achieve at most 22% accuracy in some communities, and they are releasing all artifacts and an open contribution process to expand the benchmark.
arXiv:2607. 23058v1 Announce Type: cross Abstract: Multilingual reasoning evaluation overwhelmingly relies on translating English benchmarks, a practice that introduces linguistic artifacts and fails to test culturally-grounded reasoning.
arXiv:2608.28405v1 Announce Type: new Abstract: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common...
arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.
The paper introduces TriviaRoomQA, a multilingual benchmark of 3,300 parallel multiple-choice questions across 288 topics, plus an additional 5,340 French-only questions. It evaluates 30 open-weight large language models (7–70 B parameters) from various regions, finding strong performance on knowledge-intensive subjects like history and geography but weaker results on everyday popular‑culture topics such as celebrities, music, movies, and news. The study also shows that model performance varies across languages for the same questions, indicating that factual knowledge access is not always language‑independent.
The paper compares Knowledge-Graph Based Augmentation (Graph-RAG) with Retrieval-Augmented Generation (RAG) for answering culturally specific questions. Using the LatamQA dataset, Graph-RAG, built automatically from Wikipedia via KGGen, matches RAG performance and reduces the base LLM’s error by 72% with a standard KG and 78% with a benchmark-aware variant. The approach also transfers zero‑shot to Portuguese, showing multilingual applicability.
Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics.
arXiv:2610.20419v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) models have become almost a cornerstone of open-domain question answering tasks; in which the quality of answers g...
CHARM is a multicultural benchmark that tests large language models’ ability to adopt a character’s style while respecting knowledge boundaries. It includes 40 real and fictional characters from five cultural-linguistic regions and evaluates two boundary types—Temporal and Cross-Universe—using abstention-enabled multiple-choice questions. The study finds that hallucinations mainly stem from compliance failures: models often recognize a query is out of scope yet still provide out-of-character answers, revealing systematic cultural variations in these errors.
arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.
arXiv:2508. 08879v3 Announce Type: replace-cross Abstract: The growing deployment of large language models (LLMs) across diverse cultural contexts necessitates a deeper understanding of models' hidden representations of different cultures.
arXiv:2608. 13210v1 Announce Type: cross Abstract: Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit.
arXiv:2607. 20241v1 Announce Type: cross Abstract: Culturally loaded translation poses unique challenges for machine translation (MT), as meanings are deeply embedded in socio-cultural contexts beyond surface linguistic forms.