Hugging Face Trending Papers
Your Benchmark Is an Empirical Measure Over Difficulty
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Oct 22
Your Benchmark Is an Empirical Measure Over Difficulty
The paper argues that a benchmark is more than a set of test questions; it defines an empirical distribution over difficulty. It identifies two failure modes—measure mismatch, where the benchmark’s difficulty distribution diverges from the intended one, and coverage gaps, where parts of the difficulty spectrum are missing. The authors propose diagnostic metrics with theoretical guarantees, analyze 25 benchmarks across seven domains, and introduce BenchFill, an algorithm that rewrites items to fill coverage gaps.
arXiv AI
Aug 6
Revisiting Generalization Across Difficulty Levels: It's Not So Easy
arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.
arXiv AI
2d ago
DARE: Difficulty Analysis for Repair Enhancement
DARE (Difficulty Analysis for Repair Enhancement) is a framework that measures, predicts, and interprets the difficulty of coding repair tasks using a continuous difficulty metric based on failure rates across multiple agents. It employs 151 static features to predict difficulty, analyzes contributions from seven feature groups to identify task-specific characteristics, and selects tailored agent support accordingly. Experiments on 2,594 tasks across four benchmark partitions show that DARE’s predictions correlate well with actual difficulty and that matching support to salient feature groups improves repair success.
arXiv Machine Learning
Aug 7
Predicting Task Difficulty Without Rollouts
arXiv:2608. 05797v1 Announce Type: new Abstract: Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments.
arXiv Machine Learning
Jun 5
Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning
arXiv:2603. 13761v2 Announce Type: replace Abstract: Curriculum learning--ordering training examples in a sequence to aid machine learning--takes inspiration from human learning, but has not gained widespread acceptance.
arXiv AI
Sep 21
Balance of Benchmarks: Semantic Density Reweighting for Task-Conditioned Model Comparison
The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.