The paper argues that a benchmark is more than a set of test questions; it defines an empirical distribution over difficulty. It identifies two failure modes—measure mismatch, where the benchmark’s difficulty distribution diverges from the intended one, and coverage gaps, where parts of the difficulty spectrum are missing. The authors propose diagnostic metrics with theoretical guarantees, analyze 25 benchmarks across seven domains, and introduce BenchFill, an algorithm that rewrites items to fill coverage gaps.
By Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang
arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.
By Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach
DARE (Difficulty Analysis for Repair Enhancement) is a framework that measures, predicts, and interprets the difficulty of coding repair tasks using a continuous difficulty metric based on failure rates across multiple agents. It employs 151 static features to predict difficulty, analyzes contributions from seven feature groups to identify task-specific characteristics, and selects tailored agent support accordingly. Experiments on 2,594 tasks across four benchmark partitions show that DARE’s predictions correlate well with actual difficulty and that matching support to salient feature groups improves repair success.
By Jikwang Lee, Seungwook Jung, Jaechang Nam
arXiv:2608. 05797v1 Announce Type: new Abstract: Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments.
By Stefan Krsteski, Charlotte Meyer
arXiv:2603. 13761v2 Announce Type: replace Abstract: Curriculum learning--ordering training examples in a sequence to aid machine learning--takes inspiration from human learning, but has not gained widespread acceptance.
By Amogh Inamdar, Zhenwei Tang, Ashton Anderson, Richard Zemel
The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.
By Jhen-Ke Lin, Hong-Yun Lin
arXiv:2610.28522v1 Announce Type: new
Abstract: Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make...
By Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or...
Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula.
arXiv:2608.30372v1 Announce Type: new
Abstract: As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While th...
By Minsoo Song, Chanjun Park
The paper investigates what makes software issue resolution tasks difficult for agents by proposing a measurement framework and conducting a large‑scale empirical study on the CoderForge‑Preview dataset. It extracts static features from task patches, repositories, and prompts, and uses ensemble methods, SHAP attribution, and effect size analysis to predict task outcomes. The study finds that task difficulty is largely predictable from static features (AU C = 0.863), driven mainly by patch fragmentation and repository scale, with prompt linguistic features contributing for mid‑band tasks, suggesting a layered difficulty structure.
By Ebtesam Al-Haque, Brittany Johnson
arXiv:2610.28187v1 Announce Type: cross
Abstract: Testing LLM-based software is expensive: each input needs several sampled responses, each judged by a costly oracle. Further, every prompt or model c...
By Somin Kim, Robert Feldt, Shin Yoo