arXiv Machine Learning
Jul 21
Quantifying Ranking Uncertainty in LLM Benchmarks
arXiv:2607. 16259v1 Announce Type: new Abstract: Pretrained models are typically ranked on multi-task leaderboards to assess their effectiveness across diverse tasks.
Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation
arXiv:2606. 08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts.
Related stories
arXiv Machine Learning
Jun 5
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
arXiv Machine Learning
Aug 4
Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees
arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.
Hugging Face Trending Papers
Aug 3
Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees
Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features.
arXiv Machine Learning
Jun 29
Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings
arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.
arXiv Machine Learning
Jun 4
GENEB: Why Genomic Models Are Hard to Compare
arXiv:2606. 04525v1 Announce Type: cross Abstract: Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting.
arXiv AI
Jun 24
On the Stability of Prompt Ranking in Large Language Model Evaluation
arXiv:2606. 24381v1 Announce Type: cross Abstract: Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the top-ranked one is selected for downstream use.
arXiv AI
Jun 4
Position: State-of-the-Art Claims Require State-of-the-Art Evidence
arXiv:2605. 17273v3 Announce Type: replace-cross Abstract: State-of-the-Art (SOTA) claims pervade Artificial Intelligence (AI) and Machine Learning (ML) research.
arXiv AI
Jun 8
Bounded-Abstention Pairwise Learning to Rank
arXiv:2505. 23437v2 Announce Type: replace-cross Abstract: Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts.
arXiv AI
Jun 10
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.
arXiv Machine Learning
Aug 11
A Control Function Framework for Mitigating Position Bias in Learning to Rank Systems
arXiv:2506. 06989v3 Announce Type: replace-cross Abstract: Learning-to-rank (LTR) systems commonly depend on implicit feedback, such as user clicks, because it is easy to collect and can serve as a valuable signal of user preferences.
arXiv AI
Jul 29
Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.