arXiv Computation and Language

Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities

The paper investigates how large language models (LLMs) represent nine transborder ethnic communities along China’s southwestern border with mainland Southeast Asia, such as the Sami, Rohingya, and Kurds. It introduces the concept of identity ossification, where LLMs reduce fluid, cross-border identities to a single nation-state narrative, and presents the Transborder Identity Ossification Diagnostic Framework (TIODF) to differentiate between knowledge gaps and narrative framing. Experiments with GPT‑5.1 and DeepSeek‑V3.2 in English and Chinese show that ossification rates vary widely, with DeepSeek‑V3.2’s Chinese responses exhibiting the phenomenon in 62.6% of cases, indicating that narrative framing rather than mere knowledge absence drives the issue.

arXiv AI
Sep 2

CHARM: Character Hallucination for Multicultural Role Play Benchmark

CHARM is a multicultural benchmark that tests large language models’ ability to adopt a character’s style while respecting knowledge boundaries. It includes 40 real and fictional characters from five cultural-linguistic regions and evaluates two boundary types—Temporal and Cross-Universe—using abstention-enabled multiple-choice questions. The study finds that hallucinations mainly stem from compliance failures: models often recognize a query is out of scope yet still provide out-of-character answers, revealing systematic cultural variations in these errors.

By Sunkyung Han, Nahyeon Park, Gaeun Seo, Seunghyun Yoon, JinYeong Bak
arXiv AI
Aug 20

Computational Orientalism: Measuring Structural Discourse Bias in Large Language Models Using the Middle East Cultural Sensitivity Score (MECSS)

The paper introduces the Middle East Cultural Sensitivity Score (MECSS) to quantify Orientalist bias in large language models, converting Said’s seven Orientalist operations into measurable dimensions. Using 280 conversations, it finds that GPT‑4 and Falcon3‑7B‑Instruct systematically reproduce Orientalist patterns, with Falcon scoring higher despite being regionally built. The study highlights that geographic origin alone does not mitigate bias and identifies a new failure mode, "Said‑washing," present in 87.9% of GPT‑4 interactions.

By Maha Shahid
arXiv AI
Jun 8

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.

By Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro, Mersin Konomi, Mingmeng Geng, Ahmed Asaad, Guokan Shang, Michalis Vazirgiannis
arXiv AI
Sep 10

We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation

The study investigates how a single politically charged framing term can influence large language models (LLMs) during translation tasks. By prompting eight models from Western, Chinese, and European origins to translate culturally attributed recipes across 17 languages under four framing conditions, the authors find that models resolve ambiguity rather than decline, with distinct behavior patterns tied to model families. Sensitivity to framing terms is consistent, showing that even subtle variations can modulate LLM behavior, raising concerns about implicit political judgments in translation contexts.

By Svetlana Gorovaia, Angelica Henestrosa, Ivan P. Yamshchikov
arXiv AI
Jun 2

IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

arXiv:2606. 01260v1 Announce Type: cross Abstract: Despite being home to more than 1300 ethnic groups and 700 indigenous languages, bias in Large Language Models has not been fully studied in Indonesia, thus leaving a critical gap in evaluating representational fairness and localized stereotypes within its uniquely vast, multilingual, and diverse sociocultural landscape.

By Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Fajri Koto
arXiv Computation and Language
Aug 31

Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs

The paper investigates cultural biases in large language models (LLMs) by introducing the Culture-Related Open Questions (CROQ) dataset, which contains 24‑language questions about generic culture. Experiments reveal that LLMs disproportionately favor Japan in their responses, especially when prompted in high‑resource languages, while low‑resource languages tend to highlight countries where the language is official. The study also finds that these biases emerge after supervised fine‑tuning rather than during pre‑training.

By Joseba Fernandez de Landa, Carla Perez-Almendros, Jose Camacho-Collados
arXiv Computation and Language
Sep 25

Language Specific Knowledge: Do Models Know Better in X than in English?

The paper introduces the concept of Language Specific Knowledge (LSK), showing that multilingual language models can answer certain queries better when prompted in a language other than English, sometimes even in low‑resource languages. It defines a language‑selection problem and presents several baseline methods, including the authors’ LSKExtractor, to empirically demonstrate that choosing the optimal language can improve question‑answering performance across datasets covering cultural and social norms. Experiments reveal non‑intuitive mappings, such as Gemma models excelling on Chinese and Middle Eastern topics in Spanish and Qwen models performing best on authority and responsibility queries in Arabic and Chinese.

By Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-T\"ur