The paper argues that a benchmark is more than a set of test questions; it defines an empirical distribution over difficulty. It identifies two failure modes—measure mismatch, where the benchmark’s difficulty distribution diverges from the intended one, and coverage gaps, where parts of the difficulty spectrum are missing. The authors propose diagnostic metrics with theoretical guarantees, analyze 25 benchmarks across seven domains, and introduce BenchFill, an algorithm that rewrites items to fill coverage gaps.
arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.
arXiv:2608. 05797v1 Announce Type: new Abstract: Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments.
arXiv:2603. 13761v2 Announce Type: replace Abstract: Curriculum learning--ordering training examples in a sequence to aid machine learning--takes inspiration from human learning, but has not gained widespread acceptance.
arXiv:2610.28522v1 Announce Type: new Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.
The paper investigates what makes software issue resolution tasks difficult for agents by proposing a measurement framework and conducting a large‑scale empirical study on the CoderForge‑Preview dataset. It extracts static features from task patches, repositories, and prompts, and uses ensemble methods, SHAP attribution, and effect size analysis to predict task outcomes. The study finds that task difficulty is largely predictable from static features (AU C = 0.863), driven mainly by patch fragmentation and repository scale, with prompt linguistic features contributing for mid‑band tasks, suggesting a layered difficulty structure.
arXiv:2608.30372v1 Announce Type: new Abstract: As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While th...
Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula.
arXiv:2610.28187v1 Announce Type: cross Abstract: Testing LLM-based software is expensive: each input needs several sampled responses, each judged by a costly oracle. Further, every prompt or model c...
arXiv:2607. 28634v1 Announce Type: cross Abstract: The estimation of item difficulty plays a key role in both formative assessment and large-scale high-stakes summative assessments.
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or...
The paper investigates how curriculum difficulty progression affects training in Group Relative Policy Optimization (GRPO). It finds that continually selecting medium‑difficulty samples is not optimal; a staged curriculum that gradually shifts toward harder samples improves performance, reaching a peak accuracy of 0.6591. However, progressing too far to very hard samples degrades accuracy, indicating a non‑monotonic benefit of difficulty progression.