← Back to all news
arXiv Machine Learning By Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang

Your Benchmark Is an Empirical Measure Over Difficulty

Read the original on arXiv Machine Learning →

The paper argues that a benchmark is more than a set of test questions; it defines an empirical distribution over difficulty. It identifies two failure modes—measure mismatch, where the benchmark’s difficulty distribution diverges from the intended one, and coverage gaps, where parts of the difficulty spectrum are missing. The authors propose diagnostic metrics with theoretical guarantees, analyze 25 benchmarks across seven domains, and introduce BenchFill, an algorithm that rewrites items to fill coverage gaps.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 7

Predicting Task Difficulty Without Rollouts

arXiv:2608. 05797v1 Announce Type: new Abstract: Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments.

By Stefan Krsteski, Charlotte Meyer
agentsbenchmarks
More like this →
arXiv AI
Sep 21

Balance of Benchmarks: Semantic Density Reweighting for Task-Conditioned Model Comparison

The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.

By Jhen-Ke Lin, Hong-Yun Lin
benchmarks
More like this →
arXiv AI
Aug 20

What Makes Software Issue Resolution Tasks Difficult for Agents?

The paper investigates what makes software issue resolution tasks difficult for agents by proposing a measurement framework and conducting a large‑scale empirical study on the CoderForge‑Preview dataset. It extracts static features from task patches, repositories, and prompts, and uses ensemble methods, SHAP attribution, and effect size analysis to predict task outcomes. The study finds that task difficulty is largely predictable from static features (AU C = 0.863), driven mainly by patch fragmentation and repository scale, with prompt linguistic features contributing for mid‑band tasks, suggesting a layered difficulty structure.

By Ebtesam Al-Haque, Brittany Johnson
agentsbenchmarks
More like this →