arXiv Machine Learning July 7, 2026 By Adam Fisch, Daniel Deutsch, Joshua Maynez, Alekh Agarwal, Jonathan Berant, William Cohen, Amir Globerson, Jacob Eisenstein

CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

Read the original on arXiv Machine Learning →

arXiv:2607. 05046v1 Announce Type: new Abstract: Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 30

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.

By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang

benchmarks safety

arXiv AI

Sep 18

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

The paper introduces prediction‑powered smoothing (PP‑S) and its taxonomy‑aware extension (PP‑TS) to improve point and interval estimates of domain‑specific AI performance when only a limited sample of labeled units is available. It also proposes a new design‑based cross‑validation score that is approximately unbiased for selecting between direct and smoothed estimators. Experiments on a curated benchmark and real‑world agent traffic show that the proposed methods outperform direct estimators in both accuracy and coverage, and that the new score matches the performance of an independent validation sample while providing more precise error estimates.

By Sho Kawano, Zehang Richard Li, Paul A. Parker

agents benchmarks

arXiv AI

Jun 4

Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

arXiv:2605. 23595v2 Announce Type: replace-cross Abstract: The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data.

By Trinh Pham, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen, Thanh Tam Nguyen

fine-tuning benchmarks

arXiv Machine Learning

Jul 22

How Benchmark Prediction from Fewer Data Misses the Mark

arXiv:2506. 07673v2 Announce Type: replace Abstract: Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets.

By Guanhua Zhang, Florian E. Dorner, Moritz Hardt

llms benchmarks

arXiv Machine Learning

Aug 18

Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

arXiv:2410. 13341v4 Announce Type: replace Abstract: High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem.

By Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

llms

arXiv Machine Learning

Jun 24

You Don't Need to Run Every Eval

arXiv:2606. 24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release.

By Yuchen Zeng, Dimitris Papailiopoulos

benchmarks

One email a morning, machine-written

Related stories

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

How Benchmark Prediction from Fewer Data Misses the Mark

Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

You Don't Need to Run Every Eval