The paper introduces a rubric-based benchmark to evaluate Saudi Arabic dialect and cultural competence in large language models. It comprises 31 expert-authored prompts covering idiomatic, pragmatic, lexical, and culturally embedded aspects, each paired with an expert-established ground truth. Four state-of-the-art models were scored, revealing that none exceeded 55% accuracy and that ambiguous framing was the most common error type.
By Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad, Maxim Legg
The paper introduces the Middle East Cultural Sensitivity Score (MECSS) to quantify Orientalist bias in large language models, converting Said’s seven Orientalist operations into measurable dimensions. Using 280 conversations, it finds that GPT‑4 and Falcon3‑7B‑Instruct systematically reproduce Orientalist patterns, with Falcon scoring higher despite being regionally built. The study highlights that geographic origin alone does not mitigate bias and identifies a new failure mode, "Said‑washing," present in 87.9% of GPT‑4 interactions.
By Maha Shahid
ArSciKids is a new Arabic dataset collection aimed at primary‑school science education. It contains 999 manually curated scientific‑experiment records from Saudi Arabian, Syrian, and Jordanian textbooks, and 2,167 multiple‑choice assessment questions from the Saudi Manhaji platform, all structured with detailed annotations and covering grades 1–6. The authors evaluate the datasets by fine‑tuning ten open models with LoRA, measuring generation quality with ROUGE, BLEU, and human evaluation, and plan to release the resources publicly for Arabic educational NLP research.
By Taghrid Housari, Massa Rankoussi, Jafar Abdul Majeed, Wael Tarabishi, Khloud Al Jallad, Nada Ghneim
The study audits the cultural values expressed by the decision‑only language model JEV using the 2013 Values Survey Module. By presenting 24 items to JEV under 12 matched Saudi and 12 matched American personas, in both English and Arabic, and across eight request formulations, the researchers found that JEV’s responses were highly repeatable (ICC 0.997) and that persona and language significantly influenced the model’s value profiles. Saudi personas shifted JEV’s answers toward the human Saudi‑US difference—capturing 87 % of the effect in English and 62 % in Arabic—while language, age, and gender also modulated the outcomes.
By Bushra Asseri, Abdulaziz Asseri
arXiv:2603.04410v3 Announce Type: replace-cross
Abstract: While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hi...
By Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda
Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer.
arXiv:2609.13980v1 Announce Type: cross
Abstract: Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Lea...
By Nawar S. Alseelawi, Mustafa S. Aljumaily
Camellia is a new benchmark that tests cultural bias in large language models (LLMs) across nine Asian languages and six Asian cultures. It contains 19,530 manually annotated entities linked to Asian or Western cultures and 2,173 masked social‑media contexts for these entities. Using Camellia, the authors evaluate four multilingual LLMs on cultural context adaptation, sentiment association, and entity extractive QA, finding that models struggle with cultural adaptation, exhibit differing biases across regions and families, and have difficulty understanding context in some Asian languages.
By Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo, Jaehyeok Lee, Kyungdon Lee, Lheane Marie Dizon, Mengyu Ye, Neel Kothari, Sahajpreet Singh, Sarah Masud, Tanish Patwa, Trung Thanh Tran, Zohaib Khan, Alan Ritter, Tanmoy Chakraborty, Yuki Arase, Keisuke Sakaguchi, JinYeong Bak, Wei Xu
arXiv:2610.35168v1 Announce Type: new
Abstract: The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use th...
By Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad
The paper presents an expert-driven method for turning normative principles—specifically Islamic ethical, theological, and jurisprudential traditions—into alignment data for language models. Over a year, seven experts curated 2.8K supervised fine-tuning examples and 5.4K preference pairs in Arabic-English, then evaluated models trained on these datasets. Experiments show that models trained with the curated SFT data outperform a baseline in expert judgments, while adding preference data yields a smaller, non-significant improvement.
The paper surveys the state of Explainable AI (XAI) in Arabic NLP, highlighting three gaps: a method gap where Arabic XAI relies mainly on limited post‑hoc techniques; a task gap with most work focused on classification tasks and little on generation, retrieval, or dialogue; and a linguistic gap where explanations rarely address Arabic‑specific phenomena such as morphology, dialects, and diglossia. It proposes a taxonomy of tasks, methods, linguistic units, and evaluation practices, and outlines a research agenda for linguistically grounded Arabic XAI.
By Salima Lamsiyah, Ruslan Mitkov
Qistas is a new protocol designed to evaluate how large language models treat the social, confessional, and regional constituencies of a society in a portable, pre‑registered manner. The pilot implementation focuses on Iraq and measures procedural symmetry, differential refusal, stereotype reliance, and terminological framing using paired statistical tests and a power analysis that specifies detectable effect sizes before data collection. The authors release an authoring apparatus—including a taxonomy, checklists, audit logs, and a human review notebook—and report a field test that uncovered several defect patterns common to counterfactual fairness benchmarks.
By Nawar S. Alseelawi, Mustafa S. Aljumaily