arXiv AI By Hossam Elsafty, Tobias Deu{\ss}er, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

Read the original on arXiv AI →

The paper evaluates the Jev system on Arabic, comparing it with 66 other models across 43 tasks and 9 task families. Jev 1.14.0 scores a mean task-primary metric of 69.24, slightly below Gemma-4-31B’s 71.81 but above Jev 1.13.0’s 68.77. The study also examines performance differences between Modern Standard Arabic and dialects, noting variable gaps and a notable Palestinian banking penalty for Jev 1.14.0.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
Sep 22

AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation

arXiv:2609.22796v1 Announce Type: new Abstract: Dialectal Arabic machine translation (MT) remains challenging despite recent progress in Arabic language technologies, particularly because effective t...

By Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy, Saad Ezzini, Mo El-Haj, Mustafa Jarrar, Zaid Alyafeai, Bernard Ghanem, Muhammad Abdul-Mageed
arXiv AI
Sep 1

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

The paper introduces a rubric-based benchmark to evaluate Saudi Arabic dialect and cultural competence in large language models. It comprises 31 expert-authored prompts covering idiomatic, pragmatic, lexical, and culturally embedded aspects, each paired with an expert-established ground truth. Four state-of-the-art models were scored, revealing that none exceeded 55% accuracy and that ambiguous framing was the most common error type.

By Ghassan Al-Sumaidaee, Sajjad Abdoli, Ahmed Rashad, Maxim Legg