arXiv AI
Jun 8
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding
arXiv:2606. 07167v1 Announce Type: cross Abstract: Meaningful multilingual evaluation must test models in the target language and educational context.
UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
The paper introduces UrduFactBench and UrduFactQA, two hand‑annotated benchmarks for claim verification and factual consistency evaluation in Urdu, created through a multi‑stage annotation process with native speakers. It also presents UrduFactCheck, a modular fact‑checking framework that uses both monolingual and translation‑based evidence retrieval to address the scarcity of high‑quality Urdu evidence. Experiments on twelve LLMs show that translation‑augmented pipelines outperform monolingual ones, highlighting ongoing challenges for open‑source models in Urdu.
Related stories
arXiv Computation and Language
Sep 4
Evaluating Large Language Models on Urdu Idioms
The paper introduces a new benchmark for Urdu‑to‑English idiomatic translation, featuring 4,000 manually verified sentence pairs in both native Perso‑Arabic script and Romanized Urdu. It evaluates multiple tasks—translation, paraphrasing, idiom span detection, and back‑translation—using various prompting strategies, and finds that state‑of‑the‑art large language models outperform traditional neural machine translation systems, especially in preserving figurative meaning. The study also highlights challenges posed by the lack of standardized orthography in Romanized Urdu, which affects consistency and idiom span detection.
arXiv Computation and Language
Sep 23
ARAFA: An LLM-Generated Arabic Fact-Checking Dataset
A new large-scale Arabic fact‑checking dataset called Arafa has been created using an automated pipeline that generates claims from Arabic Wikipedia, mutates them into counterfactuals, and validates them against supporting or refuting evidence. The dataset contains 181,976 claim‑evidence pairs labeled as supported, refuted, or not enough information, and human evaluation shows high inter‑annotator agreement and strong validation accuracy. Fine‑tuned transformer models on Arafa achieve a Macro F1‑score of 77%, demonstrating its usefulness for Arabic fact‑checking tasks.
arXiv Machine Learning
Sep 11
Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu
The paper investigates how multilingual large language models perform when generating stories in Urdu, a low‑resource language. The authors created a corpus of 93 Urdu stories produced by GPT‑5.1, Qwen‑3‑Max, and DeepSeek‑3.1, and manually annotated errors across a nine‑label taxonomy covering linguistic, semantic, and cultural aspects. Findings reveal frequent grammatical and semantic mistakes, lack of coherence, unnatural repetition, and pervasive cultural shallowness, with few‑shot prompting failing to resolve many of these issues.
arXiv Computation and Language
2d ago
DunbaaBERT: From Sacrifice to Semantics
DunbaaBERT is a family of Urdu RoBERTa-base models trained from scratch on a 17 GB deduplicated Urdu corpus, using Byte‑BPE vocabularies of 32k, 52k, and 96k tokens. The models are evaluated on intrinsic and downstream Urdu NLP tasks—including linguistic acceptability, news classification, offensive language detection, and sentiment analysis—showing competitive performance against strong multilingual baselines while maintaining favorable efficiency trade‑offs. The study finds that larger vocabularies do not consistently improve downstream effectiveness, with the 32k variant often delivering the best overall efficiency profile.
arXiv AI
Sep 1
Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark
arXiv:2608.30065v1 Announce Type: cross Abstract: Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains tr...
arXiv AI
Oct 15
An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution
arXiv:2610.15795v1 Announce Type: cross Abstract: Automated moderation systems are deployed at scale, yet they remain opaque to the users and moderators affected by their decisions, and their coverag...
Hugging Face Trending Papers
Aug 30
Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark
Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains transfer poorly to low-resource languages due to cult...
arXiv Computation and Language
2d ago
Turkish Multimodal Video Fact-Checking: A New Dataset and a Comparison of Zero-Shot Agentic and Fine-Tuned Systems
The paper introduces TeViD, the first multimodal Turkish video fact‑checking dataset, compiled from Teyit.org articles, translated Snopes content, and synthetic examples to address class imbalance. Two verification pipelines are built on TeViD: 3MFact‑TR, a zero‑shot system combining Qwen2.5‑7B and Qwen2‑VL‑7B with custom modules for inconsistency checking, question management, and bilingual evidence retrieval; and a fine‑tuned Qwen2.5‑7B model using LoRA for closed‑book verdicts. Both pipelines are evaluated with accuracy, macro F1, True class F1, and recall, and ablation studies are presented.
arXiv Computation and Language
Aug 28
AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification
AEScorer is an agentic evidence‑grounded framework designed for graded factuality verification, addressing the limitation of binary judgments in current methods. It operates in two stages: first, it gathers and refines external evidence through agentic search; second, it predicts a scalar factuality score to capture nuanced differences in correctness. The authors also introduce GradedVeriBench, a benchmark covering general and multi‑hop question answering, and demonstrate that AEScorer outperforms existing approaches on this new benchmark.
arXiv Computation and Language
Oct 22
AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages
AfrIFact is a new dataset designed to support automatic fact‑checking in ten African languages and English, covering information retrieval, evidence extraction, and fact verification. Experiments show that current embedding models struggle with cross‑lingual retrieval, and that cultural and news documents are easier to retrieve than healthcare documents. Large language models also lack robust multilingual fact‑verification, but few‑shot prompting and task‑specific fine‑tuning can significantly improve performance.
arXiv Computation and Language
Nov 2
TRIDENT and SHAKTI: A Framework and Benchmark for Hindi AI-Text Detection
The paper introduces SHAKTI, a large-scale Hindi benchmark for detecting AI-generated text, covering multiple genres, LLM families, prompting strategies, and adversarial transformations. It evaluates the benchmark with human annotation and examines linguistic, grammatical, and sentiment differences between human and AI texts. Building on these insights, the authors propose TRIDENT, a round‑trip translation‑based detection framework with three variants that outperforms existing baselines and demonstrates robustness across seven languages and adversarial settings.