Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,789 stories · RSS feed

arXiv AI

Jun 24

Ten Digits on a Train: AI-Assisted Verification of Two Eigenvalue Problems

arXiv:2606. 23821v1 Announce Type: cross Abstract: Accurate numerical eigenvalues are often difficult to certify, especially in singular or non-normal settings.

By Matthew J. Colbrook

benchmarks

arXiv AI

Jun 24

Probing the Misaligned Thinking Process of Language Models

arXiv:2606. 24251v1 Announce Type: new Abstract: Large language models exhibit a growing range of misaligned behaviors such as strategic deception, sandbagging, and self-preservation.

By Kaiwen Zhou, Constantin Venhoff, Jonathan Michala, Xin Eric Wang, William Saunders

llms benchmarks

arXiv Machine Learning

Jun 24

BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks

arXiv:2606. 24162v1 Announce Type: cross Abstract: Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics.

By Jin Huang, Yutong Xie, Wanli Song, Xingjian Zhang, Walter Yuan, Matthew O. Jackson, Qiaozhu Mei

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 24

Experiments with Optimal Model Trees

arXiv:2503. 12902v4 Announce Type: replace Abstract: Model trees provide an appealing way to perform interpretable machine learning for both classification and regression problems.

By Sabino Francesco Roselli, Eibe Frank

benchmarks safety

arXiv Machine Learning

Jun 24

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling

arXiv:2606. 24650v1 Announce Type: cross Abstract: We present Harmonic, a hierarchical state space model (SSM) for language modeling.

By Petr Nyoma

llms benchmarks

arXiv AI

Jun 24

Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment

arXiv:2606. 24173v1 Announce Type: cross Abstract: On-device fault detection enables real-time diagnostics without cloud dependency, but deploying machine learning models on resource-constrained hardware demands careful tradeoffs between accuracy, latency, and model size.

By Disha Patel

llms efficiency benchmarks

arXiv AI

Jun 24

Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring

arXiv:2606. 24178v1 Announce Type: cross Abstract: Pretrained vision models often misclassify inputs that are rotated, scaled, or sheared, even though these affine transformations leave the object class unchanged.

By Dominik Lindner, Johann Schmidt, Tom Siegl, Martin Becker, Sebastian Stober

benchmarks

arXiv AI

Jun 24

Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs

arXiv:2606. 24371v1 Announce Type: cross Abstract: Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.

By Stefano Mereu, Oleksandr Kuznetsov, Gabriele Marchello, Alessandro Galdelli, Emanuele Frontoni, Adriano Mancini, Ferdinando Cannella

benchmarks

arXiv AI

Jun 24

The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

arXiv:2606. 24460v1 Announce Type: cross Abstract: Commercial large language models bill, scale latency, and budget context per token.

By Olaoye Anthony Somide

llms nlp benchmarks

arXiv AI

Jun 24

Attention in Motion: Secure Platooning via Transformer-based Misbehavior Detection

arXiv:2512. 15503v3 Announce Type: replace-cross Abstract: Vehicular platooning promises transformative improvements in transportation efficiency and safety through the coordination of multi-vehicle formations enabled by Vehicle-to-Everything (V2X) communication.

By Konstantinos Kalogiannis, Ahmed Mohamed Hussain, Hexu Li, Panos Papadimitratos

llms efficiency benchmarks safety

arXiv AI

Jun 24

IPO Finance Agent: Evaluation of LLM Financial Analysts beyond Finance Agent v2, with Automated Rubric Generation -- the Case of the SpaceX (SPCX) IPO

arXiv:2606. 23032v2 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks.

By Mostapha Benhenda

llms agents benchmarks safety

arXiv AI

Jun 24

Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.

By Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, Xiao Huang

llms benchmarks

arXiv AI

Jun 24

ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms

arXiv:2512. 03476v3 Announce Type: replace-cross Abstract: Progress in computational science depends on complex numerical workflows that must faithfully encode physical laws, yet translating conceptual insight into reliable code remains a major bottleneck.

By Juan Diego Toscano, Daniel T. Chen, George Em Karniadakis

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 24

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

arXiv:2606. 24391v1 Announce Type: new Abstract: We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base.

By Arnaud Ricci

llms benchmarks safety

arXiv AI

Jun 24

ZONOS2 Technical Report

arXiv:2606. 24320v1 Announce Type: cross Abstract: We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity.

By Gabriel Clark, Sofian Mejjoute, Mohamed Osman, George Close, Beren Millidge

efficiency benchmarks

arXiv AI

Jun 24

Event-Grounded Question Answering over Long Audio via Structured Retrieval

arXiv:2602. 14612v4 Announce Type: replace-cross Abstract: Answering natural-language questions over multi-hour audio requires both event recognition and temporal grounding.

By Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada, Yinyi Guo, Erik Visser

llms rag nlp benchmarks data-engineering

arXiv Machine Learning

Jun 24

An LLM-based Two-Stage Transformer Framework for Cross-Domain Bearing Fault Diagnosis with Limited Data

arXiv:2606. 24459v1 Announce Type: new Abstract: Bearing fault diagnosis faces critical challenges when dataset heterogeneity, operating condition variations, and limited labeled data occur simultaneously in industrial environments.

By Jinghan Wang, Feng Cheng, Wentao Wu, Hang Li, Gaoliang Peng, Tianchen Liu

llms rag fine-tuning benchmarks safety

arXiv Machine Learning

Jun 24

Project Ariadne: Prompt-Conditioned Route Generation for Synthesis Planning

arXiv:2606. 24184v1 Announce Type: new Abstract: Retrosynthetic planning seeks to connect a target molecule to commercially available starting materials through a multistep route.

By Anton Morgunov, Victor S. Batista

benchmarks

arXiv Machine Learning

Jun 24

A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modeling

arXiv:2606. 24696v1 Announce Type: cross Abstract: Physics-informed surrogate models can accelerate computational fluid dynamics simulations.

By Somyajit Chakraborty, Ming Pan, Xizhong Chen

llms benchmarks

arXiv AI

Jun 24

KANLib -- A Modular, Extensible and Fast Kolmogorov-Arnold Network Implementation

arXiv:2606. 17927v2 Announce Type: replace-cross Abstract: Kolmogorov-Arnold Networks (KANs) have recently emerged as a promising alternative to traditional multilayer perceptrons by replacing linear weights with learnable univariate functions.

By Julian Hoever, Gregor Schiele

benchmarks safety

Get the daily briefing