Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI
Jul 21

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

arXiv:2602. 03061v2 Announce Type: replace-cross Abstract: Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms.

By Zihan Dong, Zhixian Zhang, Yang Zhou, Can Jin, Ruijia Wu, Linjun Zhang
arXiv Machine Learning
Jul 21

A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization

arXiv:2406. 06629v2 Announce Type: replace Abstract: This survey examines key advancements in designing features to represent optimization problem instances, algorithm instances, and their interactions within the context of single-objective continuous black-box optimization.

By Gjorgjina Cenikj, Ana Nikolikj, Ga\v{s}per Petelin, Niki van Stein, Carola Doerr, Tome Eftimov
arXiv Machine Learning
Jul 21

NIRVANA: Structured Pruning Reimagined for Large Language Model Compression

arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.

By Mengting Ai, Tianxin Wei, Sirui Chen, Jingrui He
arXiv AI
Jul 21

Do Language Models Dream of Binding Molecules? Benchmarking LLMs under Spatial Constraints

arXiv:2607. 18144v1 Announce Type: cross Abstract: Structure-based drug design (SBDD) leverages the 3D structure of protein targets, often complemented by other spatial constraints, to generate candidate binding molecules.

By Thomas MacDougall, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Maxim Malkov, Vladimir Aladinskiy, Alex Aliper, Alex Zhavoronkov
arXiv Machine Learning
Jul 21

Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables

arXiv:2603. 15802v2 Announce Type: replace Abstract: In many time series forecasting settings, the target time series is accompanied by exogenous covariates, such as promotions and prices in retail demand; temperature in energy load; calendar and holiday indicators for traffic or sales; and grid load or fuel costs in electricity pricing.

By Andres Potapczynski, Ravi Kiran Selvam, Tatiana Konstantinova, Malcolm Wolff, Kin G. Olivares, Ruijun Ma, Michael W. Mahoney, Andrew Gordon Wilson, Boris N. Oreshkin, Dmitry Efimov
arXiv Machine Learning
Jul 21

ClouDens: Operational Context-Aware Anomaly Detection for Large-scale Cloud System Monitoring

arXiv:2607. 18127v1 Announce Type: cross Abstract: With the rapid growth of cloud computing infrastructures in scale and complexity, network monitoring for Large-scale Cloud Systems (LCSs) has become increasingly challenging, requiring automated and reliable anomaly detection to maintain service availability.

By Thu T. H. Doan, Mohammad Saiful Islam, Andriy Miranskyy, Ngoc-Thanh Nguyen, Rogardt Heldal, Patrizio Pelliccione
arXiv AI
Jul 21

CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents

arXiv:2607. 16632v1 Announce Type: cross Abstract: Hardware engineering exposes coding agents to a form of long-horizon work that is difficult to capture with pass-at-k: progress is continuous, tool feedback is delayed and heterogeneous, and a backend failure may require revising RTL rather than tuning another physical-design parameter.

By Peilong Zhou, Zhirong Chen, Cangyuan Li, Haoyu Gao, Kaiyan Chang, Ziming Qu, Ying Wang
arXiv Machine Learning
Jul 21

Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs

arXiv:2603. 24002v3 Announce Type: replace Abstract: Physics-Informed Neural Networks (PINNs) for high-dimensional and high-order partial differential equations (PDEs) are primarily constrained by the $\mathcal{O}(d^k)$ spatial derivative complexity and the $\mathcal{O}(P)$ memory overhead of backpropagation (BP).

By Zhangyong Liang, Huanhuan Gao