arXiv Machine Learning June 5, 2026 By Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Read the original on arXiv Machine Learning →

arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 31

(Towards) Scalable Reliable Automated Evaluation with Large Language Models

arXiv:2607. 28282v1 Announce Type: cross Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.

By Bertil Braun, Martin Forell

llms benchmarks

arXiv Computation and Language

4d ago

Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency

Large Language Model judges are commonly used to rank texts via pairwise comparison, with reliability traditionally measured by position bias, transitivity, and pairwise agreement. This paper argues that these proxies are misleading because they are dominated by close‑rank‑gap pairs, which contribute little to the overall ranking, while far‑gap pairs carry the true ranking signal. Experiments on simulations and human‑rated corpora show weak correlation between the proxies and actual ranking accuracy, suggesting judges should be evaluated using rank‑gap‑conditional metrics against human rankings.

By Bruno Brocai, Maria Becker

llms benchmarks safety

arXiv AI

Jun 9

Margin-Adaptive Confidence Ranking for Reliable LLM Judgement

arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.

By Gaojie Jin, Yong Tao, Lijia Yu, Tianjin Huang

llms

arXiv AI

Sep 15

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluatio...

By Esma Balk{\i}r, Alice Pernthaller, Marco Basaldella, Jos\'e Hern\'andez-Orallo, Nigel Collier

llms rag benchmarks

Hugging Face Trending Papers

Aug 3

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features.

arXiv Machine Learning

Jun 15

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.

By Bora Kargi, David Salinas

llms safety