The paper argues that robustness—consistency across semantically equivalent inputs—naturally emerges once models achieve high task performance. Empirical analysis across various models and datasets shows a strong positive correlation between task difficulty and robustness, indicating that robustness is driven mainly by task-specific competence rather than inherent model properties. Consequently, as tasks mature and model performance saturates, robustness is expected to improve automatically.
arXiv:2505.13546v2 Announce Type: replace-cross Abstract: Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Exist...
The paper introduces Brittlebench, a framework and evaluation pipeline that measures how sensitive large language models (LLMs) are to prompt variations. By applying semantics‑preserving perturbations to popular benchmarks, the authors show that model performance can drop by up to 12% and that such perturbations can change model rankings in 63% of cases. The study finds that these input variations can explain up to half of the performance variance for a given model, underscoring the importance of robust evaluation methods.
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these infor...
arXiv:2608. 13566v1 Announce Type: cross Abstract: Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.
The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.
SkillCT is a new benchmark that expands agent skill assessment beyond downstream task execution to a multidimensional evaluation across nine dimensions grouped into Static, Dynamic, and Human-Assisted assessments. It measures specification quality, behavioral robustness, semantic alignment, and responsibility boundaries without requiring repeated executions. The benchmark also introduces an execution-free baseline that predicts skill quality profiles by comparing to similar reference skills, reducing token usage and evaluation time while enabling skill refinement and repository maintenance.
The paper introduces Calibration-Aware Uncertainty Cascades (CAUC), a post‑hoc framework that calibrates each model’s confidence independently and uses these calibrated scores to decide when to accept an early prediction, invoke a stronger model, or combine outputs. CAUC establishes a common reliability scale across heterogeneous models, decoupling deployment policies from specific model pools or budgets. Experiments on six language benchmarks show a 1.9% relative accuracy gain over strong‑model‑only inference while cutting strong‑model calls by about 47%, and on image classification it maintains or improves performance while reducing GFLOPs by up to 57%.
The paper introduces a new evaluation framework for confidence estimation in large language models, focusing on three properties: robustness to prompt changes, stability across semantically equivalent answers, and sensitivity to semantically different answers. It demonstrates that existing confidence estimation methods perform well on robustness and stability but often fail to detect differences in answer meaning, revealing gaps in current evaluation practices. The framework aims to guide the selection of confidence estimators for practical applications.
The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.
arXiv:2607. 27283v1 Announce Type: new Abstract: Long-horizon benchmarks often show that agents fail more as tasks become longer.