← Back to all news
arXiv Machine Learning By Harsh Kumar

Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models

Read the original on arXiv Machine Learning →

The paper investigates whether lower stereotype bias scores in multilingual language models truly indicate greater fairness. By auditing ten open models across eight languages, the authors show that lower average scores often arise from translation artifacts rather than genuine bias reduction, as item‑level correlations between English and translated scores are weak. They also identify pipeline errors that can mask bias signals and provide code to replicate their findings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Aug 27

Lower-Resource, Higher Scores: Language Bias in LLM Evaluators

The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.

By Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen
llmssafety
More like this →
arXiv Machine Learning
Oct 16

Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models

The study investigates how the calibration distribution used for Block Influence (BI) pruning of Transformer blocks affects multilingual language models. It finds that English-only calibration underestimates late‑layer importance for Indian languages, leading to suboptimal pruning decisions, while a balanced multilingual calibration mitigates perplexity loss in some models but not universally. The results highlight that pruning strategies must be validated for each architecture and language set rather than relying on a single calibration source.

By Vedant Pandhare, Piyush Dhoka
llmsefficiency
More like this →
arXiv Computation and Language
Sep 21

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

The paper investigates gender bias in machine translation evaluation metrics using an occupation-balanced subset of GAMBIT+ across seven English‑source language pairs, including a new German extension. It finds that masculine translations tend to receive higher scores and that biases align with stereotypical gender representations, though the strength varies by evaluator and language. The study highlights that assessing bias requires multiple dimensions beyond a single aggregate measure.

By Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva
nlpbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y
llmsnlpbenchmarkssafety
More like this →