Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,789 stories · RSS feed

arXiv Machine Learning

Jun 24

BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks

arXiv:2606. 24162v1 Announce Type: cross Abstract: Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics.

By Jin Huang, Yutong Xie, Wanli Song, Xingjian Zhang, Walter Yuan, Matthew O. Jackson, Qiaozhu Mei

fine-tuning benchmarks safety

arXiv AI

Jun 24

VeriPilot: An LLM-Powered Verilog Debugging Framework

arXiv:2606. 23759v1 Announce Type: cross Abstract: Verilog debugging remains one of the most time-consuming stages in digital circuit design.

By Yihan Wang, Cheng Liu, Jiazheng Zhang, Lei Zhang, Long Cheng, Xiaowei Li, Huawei Li

llms benchmarks

arXiv Machine Learning

Jun 24

Exact Schur-Sylvester Dimensionality Reductions for Non-Smooth Stochastic Complexity and Manifold Sampling

arXiv:2606. 23867v1 Announce Type: new Abstract: The exact computation of the Normalized Maximum Likelihood (NML) codelength for regular non-smooth estimators (e.

By Trenton Lau, Gary P. T. Choi

benchmarks

arXiv Machine Learning

Jun 24

DREG: A Layer-Wise Jacobian Regularization as a General-Purpose Penalty

arXiv:2606. 23942v1 Announce Type: new Abstract: We present a large-scale empirical study isolating the contributions of the Derivative Regularization penalty (DREG).

By Rowan Martnishn

llms nlp benchmarks safety

arXiv AI

Jun 24

Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs

arXiv:2606. 23938v1 Announce Type: new Abstract: Driving VLA models incorporating Chain-of-Thought (CoT) reasoning are attractive because they leverage pretrained VLM representations and expose intermediate decisions in natural language, yet current rationales often lack the step-by-step decision semantics needed to keep the rationale causally connected to the planned motion.

By Xiangbo Gao, Xiukun Huang, Boyu Lu, Junge Zhang, Mengjie Mao, Jiachen Li, Wei Xiong, Zhengzhong Tu

fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 24

LLMs are Bayesian, In Expectation, Not in Realization

arXiv:2507. 11768v3 Announce Type: replace-cross Abstract: Bayesian accounts of in-context learning face a direct objection: exact posterior predictives for exchangeable data are invariant to task-preserving order, yet transformers change next-token probabilities when the same examples are serialized differently.

By Leon Chlon, Fatima Sheaib, Zein Khamis, Maggie Chlon, Mahdi El Zein, MarcAntonio M. Awada

llms benchmarks

arXiv AI

Jun 24

Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment

arXiv:2606. 24173v1 Announce Type: cross Abstract: On-device fault detection enables real-time diagnostics without cloud dependency, but deploying machine learning models on resource-constrained hardware demands careful tradeoffs between accuracy, latency, and model size.

By Disha Patel

llms efficiency benchmarks

arXiv AI

Jun 24

Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring

arXiv:2606. 24178v1 Announce Type: cross Abstract: Pretrained vision models often misclassify inputs that are rotated, scaled, or sheared, even though these affine transformations leave the object class unchanged.

By Dominik Lindner, Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober

benchmarks

arXiv AI

Jun 24

FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation

arXiv:2507. 16696v3 Announce Type: replace-cross Abstract: Industrial signal analysis is hindered by severe data heterogeneity, which we characterize as the M5 problem.

By Pingyi Fan, Anbai Jiang, Shuwei Zhang, Xinhu Zheng, Zhiqiang Lv, Bing Han, Wenrui Liang, Junjie Li, Wei-Qiang Zhang, Yanmin Qian, Xie Chen, Jia Liu

efficiency multimodal benchmarks

arXiv AI

Jun 24

Ensemble Distributionally Robust Bayesian Optimisation with Continuous Context

arXiv:2605. 07565v2 Announce Type: replace-cross Abstract: We study Bayesian Optimisation (BO) in settings where the objective function is influenced by uncontrollable environmental contexts governed by an unknown probability distribution.

By Tigran Ramazyan, Denis Derkach

benchmarks

arXiv AI

Jun 24

Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs

arXiv:2606. 24371v1 Announce Type: cross Abstract: Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.

By Stefano Mereu, Oleksandr Kuznetsov, Gabriele Marchello, Alessandro Galdelli, Emanuele Frontoni, Adriano Mancini, Ferdinando Cannella

benchmarks

arXiv Machine Learning

Jun 24

Adversarial dynamical systems characterize when data-driven learning succeeds or fails

arXiv:2407. 06312v2 Announce Type: replace-cross Abstract: Many systems resist analytical modeling, making data-driven inference of dynamics important.

By Matthew J. Colbrook, Igor Mezi\'c, Alexei Stepanenko

benchmarks safety data-engineering

arXiv AI

Jun 24

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

arXiv:2605. 08245v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input.

By Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

llms rag agents fine-tuning multimodal benchmarks safety

arXiv AI

Jun 24

2.5-D Decomposition for LLM-Based Spatial Construction

arXiv:2605. 07066v3 Announce Type: replace Abstract: Autonomous systems that build structures from natural-language instructions need reliable spatial reasoning, yet large language models (LLMs) make systematic coordinate errors when generating three-dimensional block placements.

By Paul Whitten, Li-Jen Chen, Sharath Baddam

llms agents benchmarks

arXiv AI

Jun 24

The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

arXiv:2606. 24460v1 Announce Type: cross Abstract: Commercial large language models bill, scale latency, and budget context per token.

By Olaoye Anthony Somide

llms nlp benchmarks

arXiv Machine Learning

Jun 24

Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages

arXiv:2606. 23725v1 Announce Type: cross Abstract: Machine-learning screens for battery materials are trained and judged almost entirely against computed reference voltages, and those references carry their own systematic errors.

By Krishna Teja Vepa

benchmarks safety

arXiv Machine Learning

Jun 24

Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

arXiv:2606. 24353v1 Announce Type: cross Abstract: Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving.

By Hojun Choi, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim, Jinhan Lee

llms agents computer-vision robotics efficiency multimodal benchmarks

arXiv AI

Jun 24

GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.

By Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao

agents benchmarks

arXiv AI

Jun 24

RAVEN: A Regime-Aware Variable-context Expert Network for Financial Time Series Forecasting

arXiv:2606. 24062v1 Announce Type: cross Abstract: Financial time series forecasting presents structural challenges absent from standard benchmarks.

By Cheng He, Zhenyu Guan, Xijie Liang, Defu Lian, Jiajia Li, Enhong Chen, Patrick P. C. Lee, Geng Hu, Zehao Chen

llms benchmarks

arXiv AI

Jun 24

Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

arXiv:2606. 22068v2 Announce Type: replace-cross Abstract: Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors.

By Mathew Mithra Noel, Arindam Banerjee, Yug D. Oswal, Geraldine Bessie Amali D, Venkataraman Muthiah-Nakarajan

benchmarks

Get the daily briefing