arXiv AI June 3, 2026 By Alexander Apartsin, Yehudit Aperstein

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv Machine Learning

Aug 4

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.

By Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang

arXiv Machine Learning

Jun 5

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.

By Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

llms benchmarks

arXiv Machine Learning

Jun 15

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.

By Bora Kargi, David Salinas

llms safety

arXiv AI

Jun 9

SLMJury: Can Small Language Models Judge as Well as Large Ones?

arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.

By Anish Laddha, Nitesh Pradhan, Gaurav Srivastava

llms agents nlp benchmarks safety

Hugging Face Trending Papers

Aug 3

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features.

arXiv AI

Jun 15

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

arXiv:2606. 13685v1 Announce Type: cross Abstract: LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized.

By Abel Yagubyan

llms safety

Related stories

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

SLMJury: Can Small Language Models Judge as Well as Large Ones?

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation