← Back to all news
arXiv Computation and Language By Zihao Sun

From Language to Country to Valence: A Multilingual Audit of Stereotype Bias Across LLMs

Read the original on arXiv Computation and Language →

The paper presents a multilingual audit of stereotype bias in large language models (LLMs) using 12,000 locally written Reddit posts in English, Spanish, French, and German. It evaluates gender, racial, and religious bias across six models (three open-weight and three proprietary APIs) with a two-blank design to reduce grammatical cues, finding that religion shows the highest imbalance, Spanish leads in gender bias, and German in race and religion. The study highlights that aggregate scores can mask category- and context-specific biases, underscoring the need for detailed inspection before deploying LLMs in consequential settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 1

Evaluating and Mitigating Anti-LGBTQ Biases in German and Multilingual Language Models

The paper introduces a German-English benchmark dataset to evaluate anti‑LGBTQ biases in language models, combining community‑sourced stereotypes from German‑speaking queer individuals with a German translation of WinoQueer. Eight language models of varying sizes and architectures were assessed, revealing that they reproduce anti‑queer stereotypes with differences across identities and models. Fine‑tuning on community and progressive media content reduced bias on average, though the effect was not consistent across all models and identities.

By Melina Morch, Daniel Braun
llmsfine-tuningbenchmarkssafety
More like this →
arXiv Computation and Language
Oct 19

Cultural (In)Competence: Investigating Culture-Specific Stereotype Detection Capabilities of LLMs

arXiv:2610.22545v1 Announce Type: new Abstract: Stereotypes are subjective constructs that are mediated by the cultural context in which they are viewed. Successfully modeling them thus also requires...

By Maximilian Splieth\"over, Jannes L\"otfering, Dominik Bachmann, Alexander Espig, Willem Zuidema, Katrin Schulz, Henning Wachsmuth
llmsfine-tuning
More like this →
arXiv Computation and Language
Oct 16

MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

MBBQ (Multilingual Bias Benchmark for Question-answering) is a curated extension of the English BBQ dataset that includes Dutch, Spanish, and Turkish, designed to measure stereotypes across these languages while controlling for cultural differences and task accuracy. The dataset also provides a parallel control set to assess question-answering performance independently of bias. Experiments with several open-source and proprietary LLMs show that non‑English languages can exhibit greater bias than English, and cross‑lingual bias differences persist except in the most accurate models.

By Vera Neplenbroek, Arianna Bisazza, Raquel Fern\'andez
llmsfine-tuningbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 21

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

The paper investigates gender bias in machine translation evaluation metrics using an occupation-balanced subset of GAMBIT+ across seven English‑source language pairs, including a new German extension. It finds that masculine translations tend to receive higher scores and that biases align with stereotypical gender representations, though the strength varies by evaluator and language. The study highlights that assessing bias requires multiple dimensions beyond a single aggregate measure.

By Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva
nlpbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 3

WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities

The paper introduces WinoQueer-NL, a Dutch adaptation of the English WinoQueer benchmark, designed to assess anti‑queer bias in Dutch language models. After validating the dataset with 43 queer Dutch participants, the authors expanded it to 42,906 sentences and evaluated several Dutch and multilingual models, finding that while overall bias scores appeared neutral, specific identities—particularly transgender and non‑binary—were disproportionately favored in stereotypical sentences. The study underscores the need for culturally grounded datasets to identify and mitigate biases that affect marginalized groups in Dutch NLP systems.

By Jiska Beuk, Gerasimos Spanakis
llmsbenchmarkssafety
More like this →