arXiv AI

CHARM: Character Hallucination for Multicultural Role Play Benchmark

CHARM is a multicultural benchmark that tests large language models’ ability to adopt a character’s style while respecting knowledge boundaries. It includes 40 real and fictional characters from five cultural-linguistic regions and evaluates two boundary types—Temporal and Cross-Universe—using abstention-enabled multiple-choice questions. The study finds that hallucinations mainly stem from compliance failures: models often recognize a query is out of scope yet still provide out-of-character answers, revealing systematic cultural variations in these errors.

arXiv Computation and Language

Oct 12

Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities

The paper investigates how large language models (LLMs) represent nine transborder ethnic communities along China’s southwestern border with mainland Southeast Asia, such as the Sami, Rohingya, and Kurds. It introduces the concept of identity ossification, where LLMs reduce fluid, cross-border identities to a single nation-state narrative, and presents the Transborder Identity Ossification Diagnostic Framework (TIODF) to differentiate between knowledge gaps and narrative framing. Experiments with GPT‑5.1 and DeepSeek‑V3.2 in English and Chinese show that ossification rates vary widely, with DeepSeek‑V3.2’s Chinese responses exhibiting the phenomenon in 62.6% of cases, indicating that narrative framing rather than mere knowledge absence drives the issue.

By Minglu Sun, Lan Chen, Hang Jiang

llms safety

arXiv Machine Learning

Aug 4

Cultural Awareness is Represented but Not Decoded: Tracing Mythological Knowledge across 18 Open-Source LLMs

arXiv:2608. 02486v1 Announce Type: cross Abstract: Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently.

By Iaroslav Chelombitko, Ekaterina Chelombitko, Mika H\"am\"al\"ainen

llms

arXiv Computation and Language

Oct 15

Analyzing performance of Retrieval Augmented Generation models for Question Answering Tasks

arXiv:2610.20419v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) models have become almost a cornerstone of open-domain question answering tasks; in which the quality of answers g...

By Fazle Mohammed Tasfiq, Sabab Attin, Sheikh Md. Tahmid Hossain, Raiat Reza, Tanzim Ahmmed, Nafis Chowdhury, Anika Ahmed, Moinul Haque, Farig Yousuf Sadeque

llms rag nlp

arXiv Computation and Language

Aug 31

CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

arXiv:2608.28405v1 Announce Type: new Abstract: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common...

By Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan, Bich Ngoc Doan, Jia Wang Peh, Xiaoyuan Yi, Jing Yao, Xing Xie, Nancy F. Chen, Zhengyuan Liu, JinYeong Bak, Wafi Shamdi, Soo Kai Chie, Liew Yu Siong, Aina Azyyati Binti Mohamad Rezal, Lew Yan Yan Vanessa, Huadan Wu, Dylan Raharja, Nadya Yuki Wangsajaya, Akane Fukushige, Kazushi Kato, Koji Inoue, Tatsuya Kawahara, Jaehyung Seo, Dongjun Kim, Seungyoon Lee, Zi Haur Pang, Rui Yang Tan, Charibeth Ko Cheng, Maria Regina Justina Estuar, Jann Railey Montalan, Pham Minh Duc, Roy Ka-Wei Lee

llms fine-tuning benchmarks safety

arXiv Computation and Language

Oct 22

CulturalRiddles: A Multicultural Benchmark of Riddles

CulturalRiddles is a new benchmark consisting of original riddles created by a community effort, covering 62 cultures across 45 countries and 51 languages. The riddles are deeply rooted in their respective cultures and require abductive, compositional reasoning, factual and social knowledge, language skills, and an understanding of culturally specific associations. The authors evaluated 28 large language model configurations, finding that even state‑of‑the‑art models achieve at most 22% accuracy in some communities, and they are releasing all artifacts and an open contribution process to expand the benchmark.

By My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, Jos\'{e} Mat\'{i}as Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Neboj\v{s}a Damnjanovi\'{c}, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dhol\'{e}, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier Garc\'{i}a Gilabert, \'{A}lvaro Gonz\'{a}lez-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Joki\'{c}, Ulugbek Khalilov, Dominik Krzemi\'{n}ski, Roy Ka-Wei Lee, Haichang Li, Chen Cecilia Liu, Zhengyuan Liu, Cedric Lothritz, Houman Mehrafarin, Antonio Miranda-Escalada, Fengran Mo, Mehrnaz Mofakhami, Wafaa Mohammed, Felipe Moreno-Vera, Prahitha Movva, Jannatul Nayem, Van Ngo, Joy Olusanya, Marc P\`{a}mies, Avinash Kuney Pandey, Yurii Paniv, Fred Philippy, Vladislav Poritski, Dikshya Poudel, Joanna Rado{\l}a, Sara Rajaee, Swati Rajwal, Prerana Rane, Muhammad Ravi Shulthan Habibi, Lucas Resck, Shubhashis Roy Dipta, Ananya Sahu, Eduardo S\'{a}nchez, Borja S\'{a}nchez Clemente, Aleix Sant, Nobin Sarwar, Istiak Shihab, Santhoshini Pranathi Singaraju, Jane Smith, David Stap, Emily Z Su, Ananth K Suresh, Sajag Swami, Ruiyang Tan, Nazia Tasnim, Khalid Tounoussi, Patteera Triamamornwooth, Muhammad Saad Umar, Shankar Venkitachalam, Oksana Volchek, Lingbo Wang, Debela Desalegn Yadeta, Rushikesh Zawar, Jinghan Zhang, Mike Zhang, Weihua Zheng, Marzieh Fadaee, Akhil Arora, K\'{a}roly Boczka, Julia Kreutzer

llms benchmarks

arXiv AI

Oct 21

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

arXiv:2610.28066v1 Announce Type: cross Abstract: Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when dif...

By Haonan Huang, Yuwei Lu, Chengkun Cai

llms

arXiv AI

Jun 9

XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad

arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.

By Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou

llms nlp benchmarks

arXiv Computation and Language

Sep 10

Do LLMs Make More Mistakes If They Do Not Believe the Input Data?

The study investigates whether large language models (LLMs) are more prone to errors when they doubt the plausibility of input data, a phenomenon termed context‑memory conflict. Using non‑English and low‑resource language datasets, the authors generate text from factual, counterfactual, and fictional RDF triples in English, Czech, Slovak, and Upper Sorbian, and evaluate faithfulness with both human annotations and an LLM judge (Kimi K3). Contrary to expectations, the results show only a weak context‑memory conflict: counterfactual inputs receive slightly lower faithfulness scores than factual ones, and the choice of LLM judge can significantly affect perceived conflict strength.

By Peter Kochelka, Ale\v{s} Manuel Pap\'a\v{c}ek, Vojt\v{e}ch Dvo\v{r}\'ak, Ond\v{r}ej Du\v{s}ek

llms rag

arXiv AI

Aug 18

CulTrace: Tracing Internal Cultural Reasoning in Large Language Models

arXiv:2508. 08879v3 Announce Type: replace-cross Abstract: The growing deployment of large language models (LLMs) across diverse cultural contexts necessitates a deeper understanding of models' hidden representations of different cultures.

By Haeun Yu, Arnav Arora Seogyeong Jeong, Nadav Borenstein, Siddhesh Pawar, Jisu Shin, Jiho Jin, Junho Myung, Alice Oh, Isabelle Augenstein

llms safety

arXiv Computation and Language

Oct 19

Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation

arXiv:2605.12515v3 Announce Type: replace Abstract: Multilingual large language models (MLLMs) often answer culturally ambiguous questions inconsistently across languages, even for a fixed user perso...

By Lucas Resck, Isabelle Augenstein, Anna Korhonen

llms safety

arXiv AI

Jun 8

The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.

By Yang Zhang, Xiao Fei, Amr Mohamed, Sarah Almeida Carneiro, Mersin Konomi, Mingmeng Geng, Ahmed Asaad, Guokan Shang, Michalis Vazirgiannis

llms benchmarks

arXiv Computation and Language

Oct 27

Do Language Models Understand Igbo Culture? Beyond Multiple Choice

arXiv:2610.35168v1 Announce Type: new Abstract: The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use th...

By Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad

llms benchmarks safety

Related stories