The paper introduces the first large‑scale benchmark dataset of Bangla idioms, along with a synthetic multiple‑choice question set for idiom meaning identification. It evaluates recent large language models on three idiom‑related tasks—paraphrasing, idiom span detection, and meaning identification—using zero‑shot and few‑shot prompting. Results show significant variability across models, with Phi‑4‑mini‑instruct best at paraphrasing, Kimi‑K2‑32b‑instruct excelling at span detection, and Gemini‑2.5‑flash leading in meaning identification.
By Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi, Swakkhar Shatabda
The paper investigates how multilingual large language models perform when generating stories in Urdu, a low‑resource language. The authors created a corpus of 93 Urdu stories produced by GPT‑5.1, Qwen‑3‑Max, and DeepSeek‑3.1, and manually annotated errors across a nine‑label taxonomy covering linguistic, semantic, and cultural aspects. Findings reveal frequent grammatical and semantic mistakes, lack of coherence, unnatural repetition, and pervasive cultural shallowness, with few‑shot prompting failing to resolve many of these issues.
By Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad
The paper introduces UrduFactBench and UrduFactQA, two hand‑annotated benchmarks for claim verification and factual consistency evaluation in Urdu, created through a multi‑stage annotation process with native speakers. It also presents UrduFactCheck, a modular fact‑checking framework that uses both monolingual and translation‑based evidence retrieval to address the scarcity of high‑quality Urdu evidence. Experiments on twelve LLMs show that translation‑augmented pipelines outperform monolingual ones, highlighting ongoing challenges for open‑source models in Urdu.
By Sarfraz Ahmad, Hasan Iqbal, Momina Ahsan, Numaan Naeem, Muhammad Ahsan Riaz Khan, Arham Riaz, Muhammad Arslan Manzoor, Yuxia Wang, Preslav Nakov
arXiv:2610.35261v1 Announce Type: new
Abstract: Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whet...
By Malik Marmonier, Beno\^it Sagot, Rachel Bawden
arXiv:2606. 07167v1 Announce Type: cross Abstract: Meaningful multilingual evaluation must test models in the target language and educational context.
By Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal, Owais Aijaz, Momina Ahsan, Preslav Nakov
arXiv:2608.30065v1 Announce Type: cross
Abstract: Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains tr...
By Abdullah Hashmat, Usman Naseem, Agha Ali Raza