arXiv Machine Learning July 21, 2026 By Lei Shi, Anlan Zhang, Rita Lyu, Zhengmian Hu, Tong Yu, David Arbour, Avi Feller, Saayan Mitra, Ritwik Sinha

BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges

Read the original on arXiv Machine Learning →

arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 4

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.

By Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang

Hugging Face Trending Papers

Aug 3

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features.

arXiv Machine Learning

Jun 26

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

arXiv:2606. 26429v1 Announce Type: new Abstract: Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions.

By Aaron J. Li, Hao Huang, Youngmin Park, Yitong Ma, Wei-Lin Chiang, Li Chen, Cho-Jui Hsieh, Bin Yu, Ion Stoica

llms benchmarks

arXiv AI

Jun 2

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

arXiv:2605. 02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation.

By Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

agents benchmarks safety

arXiv AI

Jun 30

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.

By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang

benchmarks safety

arXiv AI

Jul 23

Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence

arXiv:2604. 14575v3 Announce Type: replace-cross Abstract: Marketing research often relies on parameters estimated from costly human-generated data, such as conjoint survey responses, purchase decisions, and field experiment outcomes.

By Cheng Lu, Mengxin Wang, Dennis J. Zhang, Heng Zhang

llms benchmarks

Related stories

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence