← Back to all news
arXiv AI By Wenda Xu, Sweta Agrawal, Vil\'em Zouhar, Markus Freitag, Daniel Deutsch

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

Read the original on arXiv AI →

The paper investigates the issue of self‑bias when large language models (LLMs) generate and evaluate their own benchmarks. Using machine translation as a testbed, it shows that LLMs as both test‑set creators and evaluators produce model‑specific, homogeneous outputs that inflate their own scores, even when diversity controls are applied. The bias is strong enough that each model ranks itself first, overriding peer consensus, and the phenomenon also appears in open‑ended generation tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 28

Self-Generated Text Recognition: Quality Heuristics, Cross-Task Transfer, and Downstream Bias in LLM Evaluation

The paper investigates Self‑Generated Text Recognition (SGTR), the ability of large language models (LLMs) to identify their own outputs. By evaluating 13–21 models across 6 experimental designs, it shows that SGTR accuracy varies with evaluation format, conversation structure, and task domain, and that a quality‑heuristic bias dominates results. The study also finds that fine‑tuning for SGTR in one setting can generalize to others and may cause models to prefer their own outputs when judging, highlighting potential safety concerns.

By Jesse St. Amand, Callum Canavan, Sohaib Imran, Joseph Hewson, Aaron Lutz, Shi Feng, Puria Radmard, Lennie Wells
llmsnlpfine-tuningsafety
More like this →
Hugging Face Trending Papers
Jun 25

Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores.

llmsnlpbenchmarks
More like this →