Camellia is a new benchmark that tests cultural bias in large language models (LLMs) across nine Asian languages and six Asian cultures. It contains 19,530 manually annotated entities linked to Asian or Western cultures and 2,173 masked social‑media contexts for these entities. Using Camellia, the authors evaluate four multilingual LLMs on cultural context adaptation, sentiment association, and entity extractive QA, finding that models struggle with cultural adaptation, exhibit differing biases across regions and families, and have difficulty understanding context in some Asian languages.
By Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, Tanmoy Chakraborty, Yuki Arase, Keisuke Sakaguchi, JinYeong Bak, Wei Xu
arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.
By Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou
The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.
By Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi
The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.
By Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram
arXiv:2608.03446v2 Announce Type: replace
Abstract: Multilingual large language models (LLMs) have been shown to perform better on non-English classification tasks when the representations of the giv...
By Adnan Al Ali, Kathy H\"ammerl, Jind\v{r}ich Libovick\'y, Alexander Fraser
The paper audits 51 multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented, with low-resource languages evaluated on far fewer tasks than high-resource ones. The study also highlights that most datasets rely on English translations, creating artifacts and limiting cultural relevance, and that benchmark contamination through translation leakage and corpus overlap undermines rigor. Recommendations call for natively constructed, culturally grounded, contamination-aware evaluation frameworks that truly serve the communities they aim to assess.
arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.
By Sing Hieng Wong, Hassan Sajjad, A. B. Siddique
arXiv:2509. 08022v3 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often overlook cultural and demographic variation.
By Yao Liang, Dongcheng Zhao, Feifei Zhao, Guobin Shen, Yuwei Wang, Dongqi Liang, Yi Zeng
CoCoA (Context-Conditional Cultural Alignment) is a framework designed to mitigate cultural bias in large language models by learning context-conditional behavior. It trains on entity pairs under both culturally cued and neutral contexts, using a contrastive alignment objective combined with calibration, drift regularization, and goal-aware gradient reconciliation. Evaluations on CAMeL and Camellia across ten languages and four LLMs show that CoCoA reduces the Cultural Bias Score from 43 to 24 on average while keeping near-neutral preferences at 50.2, with minimal impact on general performance.
By Kyungdon Lee, Wei Xu, Alan Ritter, Dong-Ho Lee, JinYeong Bak
arXiv:2609.25298v1 Announce Type: new
Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely...
By Yusser Al Ghussin, Eva Gavaller, Cristina Espa\~na-Bonet, Josef van Genabith, Simon Ostermann
The paper introduces the concept of Language Specific Knowledge (LSK), showing that multilingual language models can answer certain queries better when prompted in a language other than English, sometimes even in low‑resource languages. It defines a language‑selection problem and presents several baseline methods, including the authors’ LSKExtractor, to empirically demonstrate that choosing the optimal language can improve question‑answering performance across datasets covering cultural and social norms. Experiments reveal non‑intuitive mappings, such as Gemma models excelling on Chinese and Middle Eastern topics in Spanish and Qwen models performing best on authority and responsibility queries in Arabic and Chinese.
By Ishika Agarwal, Nimet Beyza Bozdag, Dilek Hakkani-T\"ur
arXiv:2609.00565v1 Announce Type: cross
Abstract: Cultural fine-tuning has become the de facto paradigm for building culture-aware large language models (LLMs), yet existing optimization exclusively...
By Jingshen Zhang, Shaoyang Xu, Wenxuan Zhang