arXiv AI September 1, 2026 By Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad, Maxim Legg

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

The paper introduces a rubric-based benchmark to evaluate Saudi Arabic dialect and cultural competence in large language models. It comprises 31 expert-authored prompts covering idiomatic, pragmatic, lexical, and culturally embedded aspects, each paired with an expert-established ground truth. Four state-of-the-art models were scored, revealing that none exceeded 55% accuracy and that ambiguous framing was the most common error type.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Sep 16

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test wha...

By Enes Altinisik, Hamdy Mubarak, Masoomali Fatehkia, Husrev_Taha_Sencar Husrev Taha Sencar

llms fine-tuning benchmarks safety

Hugging Face Trending Papers

Jul 22

HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer.

llms nlp benchmarks safety

arXiv Computation and Language

Sep 22

AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation

arXiv:2609.22796v1 Announce Type: new Abstract: Dialectal Arabic machine translation (MT) remains challenging despite recent progress in Arabic language technologies, particularly because effective t...

By Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy, Saad Ezzini, Mo El-Haj, Mustafa Jarrar, Zaid Alyafeai, Bernard Ghanem, Muhammad Abdul-Mageed

rag nlp

arXiv AI

Sep 15

Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

arXiv:2609.13980v1 Announce Type: cross Abstract: Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Lea...

By Nawar S. Alseelawi, Mustafa S. Aljumaily

llms benchmarks safety

arXiv AI

Sep 30

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

arXiv:2603.04410v3 Announce Type: replace-cross Abstract: While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hi...

By Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda

llms nlp benchmarks safety

arXiv AI

Sep 24

Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?

arXiv:2609. 28245v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored.

By AbdulRahman A. Morsy (Department of Computer Science, School of Engineering and Applied Sciences, George Washington University, Washington DC, United States), Aya Zirikly (Department of Computer Science, School of Engineering and Applied Sciences, George Washington University, Washington DC, United States, Center for Speech and Language Processing, Whiting School of Engineering, Johns Hopkins University, Baltimore MD, United States)

llms safety

One email a morning, machine-written

Related stories

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation

Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?