← Back to all news
arXiv Machine Learning By Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad

Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu

Read the original on arXiv Machine Learning →

The paper investigates how multilingual large language models perform when generating stories in Urdu, a low‑resource language. The authors created a corpus of 93 Urdu stories produced by GPT‑5.1, Qwen‑3‑Max, and DeepSeek‑3.1, and manually annotated errors across a nine‑label taxonomy covering linguistic, semantic, and cultural aspects. Findings reveal frequent grammatical and semantic mistakes, lack of coherence, unnatural repetition, and pervasive cultural shallowness, with few‑shot prompting failing to resolve many of these issues.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 4

Evaluating Large Language Models on Urdu Idioms

The paper introduces a new benchmark for Urdu‑to‑English idiomatic translation, featuring 4,000 manually verified sentence pairs in both native Perso‑Arabic script and Romanized Urdu. It evaluates multiple tasks—translation, paraphrasing, idiom span detection, and back‑translation—using various prompting strategies, and finds that state‑of‑the‑art large language models outperform traditional neural machine translation systems, especially in preserving figurative meaning. The study also highlights challenges posed by the lack of standardized orthography in Romanized Urdu, which affects consistency and idiom span detection.

By Muhammad Farmal Khan, Mousumi Akter
llmsnlpbenchmarks
More like this →
arXiv Computation and Language
Aug 28

Which India Survives Translation? Narrative Homogenisation Across Indian Oral Traditions in LLMs

The study investigates how large language models (LLMs) handle diverse Indian oral traditions, using the Rajasthani Pabuji epic, Tamil Sangam poetry, and Bengali folk tales as case studies. By prompting Claude Sonnet and Gemini with 54 generation requests across generic, culturally specific, and regional-language prompts, the authors measured reference drift and cross-tradition convergence using Sentence‑BERT embeddings. Results show that while outputs stay closer to their own tradition than to others, there is significant cross‑tradition similarity (0.52–0.66), indicating partial homogenisation; moreover, regional‑language prompting consistently reduced fidelity to authentic traditions.

By Paarth Singh Rathore
llmsrag
More like this →
arXiv AI
Aug 26

'Ghaib in Translation' aka Unseen Harm: Measuring Cross-Script Safety Inconsistency with 'Missed-in-Urdu' Scores in LLM Hate Speech Detection

The paper "Ghaib in Translation" investigates how large language models (LLMs) handle Urdu, a widely spoken language that is largely absent from safety evaluations. Five prominent LLMs—GPT‑4o, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen‑2.5, and Llama‑3.1—were tested on six datasets covering Nastaliq Urdu, Roman Urdu, English, and code‑switched Urdu‑English. The study found significant label instability between original‑script and English‑translation classifications, with missed‑in‑Urdu rates ranging from 2.4% to 9.9% (median 4.3%). A review of 205 papers across nine ALW/WOAH editions revealed no dedicated Urdu research, underscoring the language’s neglect in current safety research.

By Fawzia Zehra (Fuzzy), Kara-Isitt, Sonal Khosla, Stephen Swift
llmssafety
More like this →