Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 30

Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.

By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman

llms benchmarks safety

arXiv AI

Jun 30

FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration

arXiv:2606. 28654v1 Announce Type: cross Abstract: Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods.

By Thiru Thillai Nadarasar Bahavan, Sachith Seneviratne, Saman Halgamuge

benchmarks

arXiv Machine Learning

Jun 30

A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

arXiv:2606. 29719v1 Announce Type: new Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it.

By Liu Zewen

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 30

Residual-Guided Dictionary Learning for Spectrally Accurate Koopman Approximation

arXiv:2606. 29083v1 Announce Type: cross Abstract: Koopman theory promises linear structure in nonlinear dynamics, but numerical Koopman spectra are easy to compute and hard to trust.

By George Coote, Matthew J. Colbrook

benchmarks

arXiv Machine Learning

Jun 30

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

arXiv:2606. 29975v1 Announce Type: new Abstract: Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts.

By Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt

efficiency benchmarks

arXiv AI

Jun 30

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

arXiv:2606. 29445v1 Announce Type: cross Abstract: Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks.

By Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

llms agents nlp multimodal benchmarks

arXiv AI

Jun 30

Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification

arXiv:2606. 28409v1 Announce Type: cross Abstract: Software-compilable C programs routinely fail to complete the four-stage pipeline of a high-level synthesis (HLS) toolchain -- compilation, C simulation (CSim), synthesis, and C/RTL co-simulation (CoSim) -- because HLS accepts only a synthesizable subset of C (HLS-C).

By Zhe Zhao, Hongbing Lang, Zhihan Xiao, Luke Ztz Hu, John Imoleayo Adebisi, Songping Mai

llms rag agents benchmarks

arXiv AI

Jun 30

Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localization

arXiv:2606. 30576v1 Announce Type: cross Abstract: Cross-view object geo-localization (CVOGL) aims to locate a target object from a query view (e.

By Liyao Wang, Ruipu Wu, Haojun Xu, Lei Shi, Linjiang Huang, Si Liu

computer-vision robotics multimodal benchmarks

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

arXiv AI

Jun 30

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

arXiv:2606. 28719v1 Announce Type: new Abstract: Test-time adaptation (TTA) of vision-language models (VLMs) is essential for their robust deployment in dynamic, real-world environments.

By Guanglong Sun, Shuang Cui, Bo Lei, Liyuan Wang, Zihan Zhai, Hongwei Yan, Hang Su, Jun Zhu, Yi Zhong

llms multimodal benchmarks

arXiv AI

Jun 30

When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration

arXiv:2606. 20724v2 Announce Type: replace Abstract: Long-horizon web agents often fail in ways hidden by final-answer evaluation: they may visit useful pages, produce a well-formed answer, and terminate confidently while still missing fields, over-including unsupported items, or relying on stale evidence.

By Aagam Sogani, Botao Rui, Swetha Vaidyanathan, Rishi Agarwal, Minghao Yan, Shivaram Venkataraman

llms agents benchmarks

arXiv AI

Jun 30

MedDiffuseMix: Preserving Diagnostic Evidence with Saliency-Aware Diffusion Medical Image Data Augmentatio

arXiv:2606. 28419v1 Announce Type: cross Abstract: Limited data availability, class imbalance, and domain variability remain major barriers to reliable medical image classification.

By Teerath Kumar, Raja Vavekanand, Muhammad Turab

llms diffusion computer-vision benchmarks

arXiv AI

Jun 30

Fast and Accurate Outlier-Aware LiDAR Super-Resolution for SLAM Applications

arXiv:2606. 28607v1 Announce Type: cross Abstract: This work tackles the challenge of enhancing low-resolution LiDAR sensors for SLAM applications through a novel Deep Unrolling-based Super-Resolution (SR) model.

By Christos Anagnostopoulos, Alexandros Gkillas, Nikos Piperigkos, Aris S. Lalos

computer-vision benchmarks

arXiv AI

Jun 30

MARS: A neurosymbolic approach for interpretable drug discovery

arXiv:2410. 05289v4 Announce Type: replace Abstract: Background: Neurosymbolic (NeSy) artificial intelligence describes the combination of logic or rule-based techniques with neural networks.

By Lauren Nicole DeLong, Yojana Gadiya, Paola Galdi, Jacques D. Fleuriot, Daniel Domingo-Fern\'andez

benchmarks safety

arXiv AI

Jun 30

Instance-Conditioned Adaptation for Large-scale Generalization of Neural Routing Solver

arXiv:2405. 01906v3 Announce Type: replace Abstract: In modern intelligent transportation systems (ITS), particularly in freight transportation and logistics, real-time route planning is crucial.

By Changliang Zhou, Xi Lin, Zhenkun Wang, Xialiang Tong, Mingxuan Yuan, Qingfu Zhang

efficiency benchmarks

arXiv Machine Learning

Jun 30

Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory

arXiv:2606. 28876v1 Announce Type: cross Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory.

By Junyi Zou, Avrova Donz

llms benchmarks

arXiv AI

Jun 30

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

arXiv:2603. 03915v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs).

By Ji-Lun Peng, Yun-Nung Chen

llms agents benchmarks

arXiv Machine Learning

Jun 30

GLIP: Graph and LLM Joint Pretraining for Graph-Level Tasks

arXiv:2606. 29773v1 Announce Type: new Abstract: Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine.

By Haoxin Sun, Yiqing Lin, Yajun Huang, Chenhui Dong, Mingjun Li, Zhongzhi Zhang

llms diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

Geometric Algebra Meets Cartesian Tensors: Higher-Order Equivariance for Interatomic Potentials

arXiv:2606. 29584v1 Announce Type: cross Abstract: $\mathrm{Cl}(3,0)$ interatomic potentials, despite their algebraic elegance, predict force magnitudes accurately but force directions poorly.

By Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban

benchmarks

arXiv AI

Jun 30

OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

arXiv:2407. 19633v4 Announce Type: replace Abstract: Optimization problems are pervasive in sectors from manufacturing and distribution to healthcare.

By Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei, Connor Lawless, Madeleine Udell

llms fine-tuning benchmarks

Get the daily briefing