WikiOriginQA is a newly introduced QA dataset that is automatically constructed by using the creation dates of different Wikipedia language editions to represent the origins of knowledge. The dataset allows for the analysis of cultural bias in large language models, with experiments showing a bias toward Anglophone cultures and highlighting the importance of knowledge propagation into the Anglosphere. This automated approach offers scalable cultural bias evaluation across diverse languages without extensive manual annotation.
arXiv:2508. 08879v3 Announce Type: replace-cross Abstract: The growing deployment of large language models (LLMs) across diverse cultural contexts necessitates a deeper understanding of models' hidden representations of different cultures.
The paper investigates cultural biases in large language models (LLMs) by introducing the Culture-Related Open Questions (CROQ) dataset, which contains 24‑language questions about generic culture. Experiments reveal that LLMs disproportionately favor Japan in their responses, especially when prompted in high‑resource languages, while low‑resource languages tend to highlight countries where the language is official. The study also finds that these biases emerge after supervised fine‑tuning rather than during pre‑training.
Camellia is a new benchmark that tests cultural bias in large language models (LLMs) across nine Asian languages and six Asian cultures. It contains 19,530 manually annotated entities linked to Asian or Western cultures and 2,173 masked social‑media contexts for these entities. Using Camellia, the authors evaluate four multilingual LLMs on cultural context adaptation, sentiment association, and entity extractive QA, finding that models struggle with cultural adaptation, exhibit differing biases across regions and families, and have difficulty understanding context in some Asian languages.
The paper introduces TriviaRoomQA, a multilingual benchmark of 3,300 parallel multiple-choice questions across 288 topics, plus an additional 5,340 French-only questions. It evaluates 30 open-weight large language models (7–70 B parameters) from various regions, finding strong performance on knowledge-intensive subjects like history and geography but weaker results on everyday popular‑culture topics such as celebrities, music, movies, and news. The study also shows that model performance varies across languages for the same questions, indicating that factual knowledge access is not always language‑independent.
Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics.
arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.
The paper compares Knowledge-Graph Based Augmentation (Graph-RAG) with Retrieval-Augmented Generation (RAG) for answering culturally specific questions. Using the LatamQA dataset, Graph-RAG, built automatically from Wikipedia via KGGen, matches RAG performance and reduces the base LLM’s error by 72% with a standard KG and 78% with a benchmark-aware variant. The approach also transfers zero‑shot to Portuguese, showing multilingual applicability.
CulturalRiddles is a new benchmark consisting of original riddles created by a community effort, covering 62 cultures across 45 countries and 51 languages. The riddles are deeply rooted in their respective cultures and require abductive, compositional reasoning, factual and social knowledge, language skills, and an understanding of culturally specific associations. The authors evaluated 28 large language model configurations, finding that even state‑of‑the‑art models achieve at most 22% accuracy in some communities, and they are releasing all artifacts and an open contribution process to expand the benchmark.
MBBQ (Multilingual Bias Benchmark for Question-answering) is a curated extension of the English BBQ dataset that includes Dutch, Spanish, and Turkish, designed to measure stereotypes across these languages while controlling for cultural differences and task accuracy. The dataset also provides a parallel control set to assess question-answering performance independently of bias. Experiments with several open-source and proprietary LLMs show that non‑English languages can exhibit greater bias than English, and cross‑lingual bias differences persist except in the most accurate models.
arXiv:2610.20472v1 Announce Type: new Abstract: Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute...
arXiv:2609.22494v1 Announce Type: new Abstract: In recent years, there has been a surge of interest in Cultural NLP, with substantial efforts to create globally inclusive NLP systems. The rapid growt...
arXiv:2608.21558v1 Announce Type: cross Abstract: Recent advances in LLMs and the adoption of RAG systems in industry have created a need for domain-specific question-answer datasets that can assess...