arXiv Machine Learning July 22, 2026 By Guanhua Zhang, Florian E. Dorner, Moritz Hardt

How Benchmark Prediction from Fewer Data Misses the Mark

Read the original on arXiv Machine Learning →

arXiv:2506. 07673v2 Announce Type: replace Abstract: Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 29

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.

By Rostislav Gusev, Alexey Zaytsev

nlp benchmarks

arXiv Computation and Language

6d ago

Large Language Model Selection with Limited Annotations

arXiv:2605.24981v2 Announce Type: replace Abstract: Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotati...

By Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve G\"urel

llms

arXiv AI

Jun 2

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.

By Maty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung, Christoph Bregler, Stephanie C. Y. Chan

llms benchmarks

arXiv Machine Learning

4d ago

UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

arXiv:2609.37076v1 Announce Type: new Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...

By Puning Yang, Qizhou Wang, Junchi Yu, Bo Han, Xiuying Chen

llms

arXiv AI

Jun 4

Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

arXiv:2605. 23595v2 Announce Type: replace-cross Abstract: The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data.

By Trinh Pham, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen, Thanh Tam Nguyen

fine-tuning benchmarks

arXiv Machine Learning

Jul 7

CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

arXiv:2607. 05046v1 Announce Type: new Abstract: Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development.

By Adam Fisch, Daniel Deutsch, Joshua Maynez, Alekh Agarwal, Jonathan Berant, William Cohen, Amir Globerson, Jacob Eisenstein

efficiency benchmarks

One email a morning, machine-written

Related stories

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

Large Language Model Selection with Limited Annotations

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion