arXiv:2611.07949v1 Announce Type: new Abstract: Keyword retrieval returns expressions of personal anxiety together with many other uses of anxiety-related language. We present a benchmark of 1,000 Ar...
The study compares six approaches—frontier commercial models, fine‑tuned smaller models, and conventional classifiers—for detecting anxiety in Reddit posts. It reveals a significant lexical bias: 69.3% of anxiety‑labelled posts contain the word "anxiety" or a variant, allowing models to perform well via keyword matching rather than true language understanding. After removing these terms, the frontier model still leads (F1 = 0.846), but a 110 M‑parameter domain‑adapted encoder achieves a close score (F1 = 0.831) without external API calls, and lexical dependence varies widely across models.
arXiv:2611.07926v1 Announce Type: new Abstract: Engagement features developed for English are routinely applied to Arabic, but unvocalized Arabic changes what a lexical feature measures. We introduce...
The paper introduces a rubric-based benchmark to evaluate Saudi Arabic dialect and cultural competence in large language models. It comprises 31 expert-authored prompts covering idiomatic, pragmatic, lexical, and culturally embedded aspects, each paired with an expert-established ground truth. Four state-of-the-art models were scored, revealing that none exceeded 55% accuracy and that ambiguous framing was the most common error type.
arXiv:2606. 20255v1 Announce Type: cross Abstract: We introduce the Meaning Intelligence Framework (MIF), a nine-dimension annotation and evaluation schema for Nigerian public discourse that separates surface sentiment from true communicative intent.
arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test wha...
The paper surveys the state of Explainable AI (XAI) in Arabic NLP, highlighting three gaps: a method gap where Arabic XAI relies mainly on limited post‑hoc techniques; a task gap with most work focused on classification tasks and little on generation, retrieval, or dialogue; and a linguistic gap where explanations rarely address Arabic‑specific phenomena such as morphology, dialects, and diglossia. It proposes a taxonomy of tasks, methods, linguistic units, and evaluation practices, and outlines a research agenda for linguistically grounded Arabic XAI.
Qistas is a new protocol designed to evaluate how large language models treat the social, confessional, and regional constituencies of a society in a portable, pre‑registered manner. The pilot implementation focuses on Iraq and measures procedural symmetry, differential refusal, stereotype reliance, and terminological framing using paired statistical tests and a power analysis that specifies detectable effect sizes before data collection. The authors release an authoring apparatus—including a taxonomy, checklists, audit logs, and a human review notebook—and report a field test that uncovered several defect patterns common to counterfactual fairness benchmarks.
Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer.
The study evaluates large language models for assessing suicide risk in Arabic crisis helpline calls, comparing Arabic and English models. Using de‑identified transcripts from Lebanon’s National Lifeline, the researchers fine‑tuned instruction‑tuned LLMs and transformer encoders, achieving a macro‑F1 of 81.19 and ROC‑AUC of 90.61 for high‑risk calls in Arabic, and 85.00/92.59 in English. The results show that high‑risk calls are more distinguishable than at‑risk calls, and translating to English does not degrade performance, indicating potential for operator‑facing tools.
arXiv:2609.13980v1 Announce Type: cross Abstract: Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Lea...
The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.