← Back to all news
arXiv AI By Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin, Alexander Antipov, Ruslana Raemgulova

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

Read the original on arXiv AI →

The paper introduces GYROval, a benchmark designed to assess cultural value orientation in large language models along the Inglehart‑Welzel axes across multiple domains and roles. It uses binary contrastive scenarios where both options are valid, and scores models by the proportion of responses aligning with each pole. The benchmark was tested on twenty models, with eleven also evaluated in Russian and at a different sampling temperature, and its stability was analyzed using tie‑corrected Kendall’s W against a permutation null.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 15

Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models

The paper investigates whether lower stereotype bias scores in multilingual language models truly indicate greater fairness. By auditing ten open models across eight languages, the authors show that lower average scores often arise from translation artifacts rather than genuine bias reduction, as item‑level correlations between English and translated scores are weak. They also identify pipeline errors that can mask bias signals and provide code to replicate their findings.

By Harsh Kumar
llmssafety
More like this →
Hugging Face Trending Papers
Jul 9

PLURAL: A Global Dataset for Value Alignment

Large language models (LLMs) are used worldwide, yet disproportionately reflect Western values, limiting their ability to represent diverse value systems. We introduce PLURAL, a large-scale, value-focused preference dataset grounded in the Integrated Values Survey (IVS), a nationally representative survey spanning 92 countries.

llmssafety
More like this →
arXiv Computation and Language
Oct 27

Do Language Models Understand Igbo Culture? Beyond Multiple Choice

arXiv:2610.35168v1 Announce Type: new Abstract: The value of language models for any community depends on whether a model actually understands the culture of the people it intends to serve. We use th...

By Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad
llmsbenchmarkssafety
More like this →