arXiv Computation and Language September 3, 2026 By Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend

Mediocrity is the key for LLM as a Judge Anchor Selection

Read the original on arXiv Computation and Language →

The paper examines how the choice of anchor model in LLM-as-a-judge evaluations affects reliability. By testing 22 anchors on the Arena-Hard-v2.0 dataset, it shows that extreme anchors (best or worst performers) are poor choices, reducing correlation with human rankings. The study quantifies the anchor effect size, compares it to judge model selection, and offers guidelines and power‑analysis recommendations for more reliable benchmark design.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jul 3

PACE: A Proxy for Agentic Capability Evaluation

arXiv:2607. 02032v1 Announce Type: new Abstract: Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure.

By Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig

llms agents benchmarks

arXiv Machine Learning

Jun 15

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.

By Bora Kargi, David Salinas

llms safety

arXiv Machine Learning

Jun 29

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.

By Rostislav Gusev, Alexey Zaytsev

nlp benchmarks

arXiv AI

Jun 4

Knowledge Index of Noah's Ark

arXiv:2606. 05104v1 Announce Type: new Abstract: Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consensus; and unaudited ranking instability under bounded test budgets.

By Sheng Jin, Minghao Liu, Yunze Xiao, Zeqi Zhou, Heli Qi, Yifan Yao, Meishu Song, Kaijing Ma, Xuan Zhang, Sicong Jiang, Yizhe Li, Ningshan Ma, Jie Wei, Ziniu Li, Minglai Yang, Bangya Liu, Yiming Liang, Xiao Fang, Qingcheng Zeng, Jiarui Liu, Rui Yang, Shen Yan, Wenhao Huang, Jiaheng Liu, Zihan Wang, Weihao Xuan, Ge Zhang

llms benchmarks

arXiv Machine Learning

Aug 5

Dynamically Allocating Evaluation Effort for Model Ranking

arXiv:2608. 03437v1 Announce Type: cross Abstract: While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability.

By Vil\'em Zouhar, Julia Kreutzer, Alon Lavie, Tom Kocmi, Matt Post, Ond\v{r}ej Bojar, Mrinmaya Sachan

nlp reinforcement-learning benchmarks

arXiv Machine Learning

Aug 12

When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design

arXiv:2608. 10528v1 Announce Type: cross Abstract: Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost.

By Utshab Kumar Ghosh, Shubham Chatterjee

llms rag efficiency

One email a morning, machine-written

Related stories

PACE: A Proxy for Agentic Capability Evaluation

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

Knowledge Index of Noah's Ark

Dynamically Allocating Evaluation Effort for Model Ranking

When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design