arXiv AI
Multi-Test Nation Bias Benchmarking of LLMs with Explicit Unbiased Baselines
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 10
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
arXiv:2602.02219v3 Announce Type: replace Abstract: Large language models are widely employed as evaluators, a paradigm commonly referred to as LLM-as-a-judge. Prior research has predominantly examin...
arXiv Computation and Language
4d ago
Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models
Qistas is a new protocol designed to evaluate how large language models treat the social, confessional, and regional constituencies of a society in a portable, pre‑registered manner. The pilot implementation focuses on Iraq and measures procedural symmetry, differential refusal, stereotype reliance, and terminological framing using paired statistical tests and a power analysis that specifies detectable effect sizes before data collection. The authors release an authoring apparatus—including a taxonomy, checklists, audit logs, and a human review notebook—and report a field test that uncovered several defect patterns common to counterfactual fairness benchmarks.
arXiv AI
Jul 13
Geopolitical alignment: Endorsement effects in large language models
arXiv:2607. 09262v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to summarize and evaluate policy-relevant information, but it remains unclear whether their judgments are implicitly shaped by geopolitical cues.
arXiv Machine Learning
Jul 3
Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges
arXiv:2607. 02104v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers.
arXiv AI
Jun 6
A Systematic Analysis of Biases in Large Language Models
arXiv:2512. 15792v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making.
arXiv AI
Aug 25
Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator
arXiv:2608.22432v1 Announce Type: cross Abstract: Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchma...