← Back to all news
arXiv AI By Shir Ashury-Tahan, Ariel Gera, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen

Robustness as an Emergent Property of Task Performance

Read the original on arXiv AI →

The paper argues that robustness—consistency across semantically equivalent inputs—naturally emerges once models achieve high task performance. Empirical analysis across various models and datasets shows a strong positive correlation between task difficulty and robustness, indicating that robustness is driven mainly by task-specific competence rather than inherent model properties. Consequently, as tasks mature and model performance saturates, robustness is expected to improve automatically.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 23

Brittlebench: Quantifying LLM robustness via prompt sensitivity

The paper introduces Brittlebench, a framework and evaluation pipeline that measures how sensitive large language models (LLMs) are to prompt variations. By applying semantics‑preserving perturbations to popular benchmarks, the authors show that model performance can drop by up to 12% and that such perturbations can change model rankings in 63% of cases. The study finds that these input variations can explain up to half of the performance variance for a given model, underscoring the importance of robust evaluation methods.

By Angelika Romanou, Mark Ibrahim, Candace Ross, Kerem Oktar, Chantal Shaib, Samuel J. Bell, Anaelia Ovalle, Antoine Bosselut, Jesse Dodge, Koustuv Sinha, Adina Williams
llmsbenchmarks
More like this →
arXiv AI
Aug 17

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

arXiv:2608. 13566v1 Announce Type: cross Abstract: Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.

By Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov, Mikhail Evtikhiev, Ana Terna, Rastislav Rabatin, Timur Kudashev, Timofey Bryksin, Arina Puchkova, Patrik Bartak, Egor Bogomolov, Sergey Titov
llmsagentsfine-tuningbenchmarks
More like this →
arXiv AI
Sep 21

Balance of Benchmarks: Semantic Density Reweighting for Task-Conditioned Model Comparison

The paper introduces Balance of Benchmarks (BoB), a framework that improves task-conditioned model comparison by weighting benchmark evidence based on semantic density, equating scores across varying difficulty levels, and pooling task-relevant residuals. BoB retains all eligible benchmark data while adjusting its influence, outperforming uniform averaging on the WildScores dataset with higher Spearman correlation, lower MAE, and better shortlist hit rates. The method also reduces ranking instability when benchmarks are repeated or paraphrased, and lowers retrospective regret in model selection.

By Jhen-Ke Lin, Hong-Yun Lin
benchmarks
More like this →