arXiv:2606. 07167v1 Announce Type: cross Abstract: Meaningful multilingual evaluation must test models in the target language and educational context.
By Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal, Owais Aijaz, Momina Ahsan, Preslav Nakov
The paper introduces a new benchmark for Urdu‑to‑English idiomatic translation, featuring 4,000 manually verified sentence pairs in both native Perso‑Arabic script and Romanized Urdu. It evaluates multiple tasks—translation, paraphrasing, idiom span detection, and back‑translation—using various prompting strategies, and finds that state‑of‑the‑art large language models outperform traditional neural machine translation systems, especially in preserving figurative meaning. The study also highlights challenges posed by the lack of standardized orthography in Romanized Urdu, which affects consistency and idiom span detection.
By Muhammad Farmal Khan, Mousumi Akter
A new large-scale Arabic fact‑checking dataset called Arafa has been created using an automated pipeline that generates claims from Arabic Wikipedia, mutates them into counterfactuals, and validates them against supporting or refuting evidence. The dataset contains 181,976 claim‑evidence pairs labeled as supported, refuted, or not enough information, and human evaluation shows high inter‑annotator agreement and strong validation accuracy. Fine‑tuned transformer models on Arafa achieve a Macro F1‑score of 77%, demonstrating its usefulness for Arabic fact‑checking tasks.
By Christophe Khalil, Shady Elbassuoni, Rida Assaf
The paper investigates how multilingual large language models perform when generating stories in Urdu, a low‑resource language. The authors created a corpus of 93 Urdu stories produced by GPT‑5.1, Qwen‑3‑Max, and DeepSeek‑3.1, and manually annotated errors across a nine‑label taxonomy covering linguistic, semantic, and cultural aspects. Findings reveal frequent grammatical and semantic mistakes, lack of coherence, unnatural repetition, and pervasive cultural shallowness, with few‑shot prompting failing to resolve many of these issues.
By Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad
DunbaaBERT is a family of Urdu RoBERTa-base models trained from scratch on a 17 GB deduplicated Urdu corpus, using Byte‑BPE vocabularies of 32k, 52k, and 96k tokens. The models are evaluated on intrinsic and downstream Urdu NLP tasks—including linguistic acceptability, news classification, offensive language detection, and sentiment analysis—showing competitive performance against strong multilingual baselines while maintaining favorable efficiency trade‑offs. The study finds that larger vocabularies do not consistently improve downstream effectiveness, with the 32k variant often delivering the best overall efficiency profile.
By Iffat Maab, Waleed Jamil, Raphael Schmitt
arXiv:2608.30065v1 Announce Type: cross
Abstract: Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains tr...
By Abdullah Hashmat, Usman Naseem, Agha Ali Raza