The paper investigates whether lower stereotype bias scores in multilingual language models truly indicate greater fairness. By auditing ten open models across eight languages, the authors show that lower average scores often arise from translation artifacts rather than genuine bias reduction, as item‑level correlations between English and translated scores are weak. They also identify pipeline errors that can mask bias signals and provide code to replicate their findings.
By Harsh Kumar
arXiv:2605.29637v2 Announce Type: replace
Abstract: Large language models often exhibit a substantial gap between their performance in English and in lower-resourced languages on equivalent knowledge...
By Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi, Akshay Agarwal, Jasabanta Patro
arXiv:2507.23248v2 Announce Type: replace-cross
Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across th...
By Shimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat
The study replicates a distributional‑semantics extractive summarisation method for Hindi, adapting all language‑specific components to Devanagari. Evaluated on the Hindi portions of XL‑Sum and FIRE ILSUM 2.0 with a Devanagari‑aware ROUGE scorer, the replicated system performs significantly worse than a simple three‑sentence lead baseline. Feature ablation shows that sentence position alone reproduces the lead baseline, while other features only steer extraction toward long, entity‑dense body sentences, and TextRank performs identically.
"whyItMatters":"The results indicate that current Hindi summarisation benchmarks cannot reward non‑lead content selection, highlighting the need for purpose‑built evaluation resources."
By Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh, Mohsin Altaf Wani, Abid Hussain Wani, Hilal Ahmad Khanday, Niyaz Ahmad Wani
arXiv:2604. 04532v2 Announce Type: replace-cross Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings.
By Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
By Priyansh Srivastava
arXiv:2610. 31407v1 Announce Type: new Abstract: Multilingual language models can give different identity judgments when the same people's names are written in different scripts.
By Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu, Arpit Jain, Ashwanth Krishnan
arXiv:2610.18420v1 Announce Type: cross
Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...
By Ramchand Kumaresan, Madhusudhanan G, Kaushik Kezendirane
arXiv:2608.30092v1 Announce Type: cross
Abstract: We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single...
By Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.
IndicQE-APE is a consolidated benchmark that unifies quality estimation (QE) and automatic post‑editing (APE) data for nine Indic language pairs, comprising 126,754 instances with multiple aligned labels such as direct assessment, human post‑edit, word‑level OK/BAD tags, and error explanations. The dataset includes a stratified test set across four difficulty axes and supports training and evaluation of six prompted large language models, three COMET metrics, and three APE systems. Experiments reveal that segments with conflicting holistic and token‑level quality signals are consistently ranked lower, while annotator disagreement shows no effect when controlled for score distribution.
whyItMatters":"The benchmark provides a unified resource for training and evaluating QE and APE across Indic languages, enabling consistent comparison of models and metrics on a shared dataset."
By Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare, Daria Sokova, Shenbin Qian, Girish Koushik, Tharindu Ranasinghe, Constantin Or\u{a}san, Chrysoula Zerva, Ricardo Rei, Fr\'ed\'eric Blain, Andr\'e F. T. Martins, Marco Turchi, Matteo Negri, Anoop Kunchukuttan, Mitesh M. Khapra, Pushpak Bhattacharyya
arXiv:2610.02926v1 Announce Type: new
Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multiling...
By Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth