arXiv:2605.12515v3 Announce Type: replace Abstract: Multilingual large language models (MLLMs) often answer culturally ambiguous questions inconsistently across languages, even for a fixed user perso...
arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.
The paper introduces CultureLens, a benchmark with 9,261 scenarios from 75 countries that tests how large language models (LLMs) incorporate cultural information. Experiments on Llama‑3.1‑8B, Gemma‑3‑12B, and Qwen‑3.5‑27B show that presenting a contradictory cultural norm reduces aligned answers by 32‑47 percentage points, even when the model knows the correct country. Fine‑tuning with contradictory‑norm examples improves alignment to 85‑93% under contradictory norms while only slightly affecting performance under true norms.
arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test wha...
CuBEs introduces culturally‑situated behavioral evaluations for large language models, adding cultural context to test scenarios and assessments. The authors built a human‑labeled dataset covering 12 cultures, revealing significant cross‑cultural differences that one‑size‑fits‑all judgments miss. Evaluating 13 LLMs shows that culturally situated tests uncover varied behaviors, such as Western political bias versus non‑Western religious or colonial biases, which standard evaluations overlook.
arXiv:2610.22545v1 Announce Type: new Abstract: Stereotypes are subjective constructs that are mediated by the cultural context in which they are viewed. Successfully modeling them thus also requires...
arXiv:2609.10253v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in globally used assistants, yet their default choices in culturally grounded everyday situation...
VakyArth is the first pragmatic benchmark for Indic languages, covering Hindi, Punjabi, Tamil, and Malayalam. It tests models on five pragmatic phenomena—deixis, speech acts, implicature, social pragmatics, and coherence—using multiple-choice questions, natural language inference, and translation tasks authored by native speakers. Evaluation of multilingual LLMs shows consistent failures on pragmatic meanings rooted in Indic linguistic and cultural conventions, with systematic differences across languages and tasks.
MemeCULT-1K is a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, each paired with a cultural context note and three human-written explanations, plus an additional set of 54 Bengali regional dialect memes. The study evaluates thirteen vision‑language models under meme‑only and context‑aware settings, showing that providing minimal cultural context consistently improves performance across all models and languages. Error analysis indicates closed‑source models struggle with entity and reference misidentification, while open‑source models are limited by broader cultural knowledge gaps, especially in linguistic and phonological aspects.
The paper investigates how large language models (LLMs) represent nine transborder ethnic communities along China’s southwestern border with mainland Southeast Asia, such as the Sami, Rohingya, and Kurds. It introduces the concept of identity ossification, where LLMs reduce fluid, cross-border identities to a single nation-state narrative, and presents the Transborder Identity Ossification Diagnostic Framework (TIODF) to differentiate between knowledge gaps and narrative framing. Experiments with GPT‑5.1 and DeepSeek‑V3.2 in English and Chinese show that ossification rates vary widely, with DeepSeek‑V3.2’s Chinese responses exhibiting the phenomenon in 62.6% of cases, indicating that narrative framing rather than mere knowledge absence drives the issue.
The paper investigates whether lower stereotype bias scores in multilingual language models truly indicate greater fairness. By auditing ten open models across eight languages, the authors show that lower average scores often arise from translation artifacts rather than genuine bias reduction, as item‑level correlations between English and translated scores are weak. They also identify pipeline errors that can mask bias signals and provide code to replicate their findings.
The paper introduces a rubric-based benchmark to evaluate Saudi Arabic dialect and cultural competence in large language models. It comprises 31 expert-authored prompts covering idiomatic, pragmatic, lexical, and culturally embedded aspects, each paired with an expert-established ground truth. Four state-of-the-art models were scored, revealing that none exceeded 55% accuracy and that ambiguous framing was the most common error type.