The paper introduces a German-English benchmark dataset to evaluate anti‑LGBTQ biases in language models, combining community‑sourced stereotypes from German‑speaking queer individuals with a German translation of WinoQueer. Eight language models of varying sizes and architectures were assessed, revealing that they reproduce anti‑queer stereotypes with differences across identities and models. Fine‑tuning on community and progressive media content reduced bias on average, though the effect was not consistent across all models and identities.
By Melina Morch, Daniel Braun
arXiv:2610.22545v1 Announce Type: new
Abstract: Stereotypes are subjective constructs that are mediated by the cultural context in which they are viewed. Successfully modeling them thus also requires...
By Maximilian Splieth\"over, Jannes L\"otfering, Dominik Bachmann, Alexander Espig, Willem Zuidema, Katrin Schulz, Henning Wachsmuth
MBBQ (Multilingual Bias Benchmark for Question-answering) is a curated extension of the English BBQ dataset that includes Dutch, Spanish, and Turkish, designed to measure stereotypes across these languages while controlling for cultural differences and task accuracy. The dataset also provides a parallel control set to assess question-answering performance independently of bias. Experiments with several open-source and proprietary LLMs show that non‑English languages can exhibit greater bias than English, and cross‑lingual bias differences persist except in the most accurate models.
By Vera Neplenbroek, Arianna Bisazza, Raquel Fern\'andez
The paper investigates gender bias in machine translation evaluation metrics using an occupation-balanced subset of GAMBIT+ across seven English‑source language pairs, including a new German extension. It finds that masculine translations tend to receive higher scores and that biases align with stereotypical gender representations, though the strength varies by evaluator and language. The study highlights that assessing bias requires multiple dimensions beyond a single aggregate measure.
By Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva
The paper introduces WinoQueer-NL, a Dutch adaptation of the English WinoQueer benchmark, designed to assess anti‑queer bias in Dutch language models. After validating the dataset with 43 queer Dutch participants, the authors expanded it to 42,906 sentences and evaluated several Dutch and multilingual models, finding that while overall bias scores appeared neutral, specific identities—particularly transgender and non‑binary—were disproportionately favored in stereotypical sentences. The study underscores the need for culturally grounded datasets to identify and mitigate biases that affect marginalized groups in Dutch NLP systems.
By Jiska Beuk, Gerasimos Spanakis
arXiv:2610.29896v1 Announce Type: new
Abstract: Existing social bias benchmarks evaluate large language models mostly on fluent, well-formed English, reflecting native speaker usage and leaving non-n...
By Federico Marcuzzi, Martin Christoph, Iryna Gurevych, Alla Rozovskays
The study examines how multilingual large language models (LLMs) produce outputs that differ across sociocultural contexts, highlighting that identity labels and source-language cues can mislead assessments of cultural grounding. Using a human‑validated, multi‑agent audit on 89,253 outputs from 12 LLMs in English, French, and Chinese across 18 occupations and three task conditions, the authors find that bias representation varies systematically by language and task. Removing direct identity cues reduces identity‑label prediction in English and Chinese but not in French, and the source language’s cultural context consistently receives the highest relevance score, though this signal weakens after translation or name masking.
"whyItMatters":"The findings show that surface cues can obscure true cross‑cultural patterns, underscoring the need for careful audit designs to avoid misleading conclusions about bias in multilingual LLMs."
By Yuanjun Feng, Tanzhou Liu, Stefan Feuerriegel, Yash Raj Shrestha
The paper introduces GPTBIAS, a framework that uses powerful large language models like GPT‑4 to evaluate bias in other LLMs. It employs specially crafted prompts called Bias Attack Instructions to probe for bias and outputs a bias score along with detailed information such as bias types, affected demographics, keywords, reasons, and improvement suggestions. Extensive experiments demonstrate the framework’s effectiveness and usability.
By Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, Mykola Pechenizkiy
arXiv:2604. 01925v2 Announce Type: replace-cross Abstract: Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly.
By Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru, Abhijnan Chakraborty
arXiv:2603. 13891v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring.
By Petter T\"ornberg
The study evaluates name-based bias in six local large language models using a fill-in-the-blank task with first names from U.S. Social Security and a race classification dataset. Across 300,000 responses, the models consistently favored names that were less stereotypically male, older, popular, and white, with gender bias being the strongest effect. The bias varied with sentence context but remained significant, underscoring the persistence of name bias in local LLMs.
By Kalyan Bhetwal, Ceren Mert, Ian Whitehouse
Names carry demographic indicators such as age, gender, and race, which makes them a subtle but strong target of bias in large language models (LLMs). In this work, we present an evaluation of name-ba...