Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,750 stories · RSS feed

arXiv AI
Jul 7

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

arXiv:2607. 03487v1 Announce Type: cross Abstract: Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored.

By Alberto Foresti, Ivan Butakov, Alexander Tolmachev, Giulio Franzese, Alexey Frolov, Pietro Michiardi
arXiv AI
Jul 7

Graph Unitary Message Passing

arXiv:2403. 11199v2 Announce Type: replace-cross Abstract: Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the graph propagation operator.

By Haiquan Qiu, Quanming Yao
arXiv AI
Jul 7

LLMoxie: Exploring Agentic AI for Scientific Software Development

arXiv:2607. 02703v1 Announce Type: cross Abstract: In this paper, we describe LLMoxie, an institutional AI platform whose three-tiered architecture supports multi-cloud and on-premise inference, a LiteLLM/MLflow control plane for authentication, budgeting, PII masking, and observability, and an application augmentation layer for AI coding agents.

By Landung Setiawan, Anant Mittal, Cordero Core, Anshul Tambay, Carlos Garcia Jurado Suarez, David A. C. Beck, Andrew J. Connolly, Vani Mandava
arXiv Machine Learning
Jul 7

Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Control

arXiv:2607. 04978v1 Announce Type: new Abstract: Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels, but every existing JEPA world model commits at training time to a single inference paradigm: either trajectory optimisation in a learned dynamics model, or direct behaviour cloning.

By Ruslan Rakhimov, George Bredis, Yuriy Maksyuta, Daniil Gavrilov
arXiv AI
Jul 7

RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

arXiv:2607. 04729v1 Announce Type: cross Abstract: LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace.

By Tarek Elsayed, Shiping Yang, Eunsong Koh, Sanika Goyal, Vincent Huang, Paul Ngo, Nathan Young, Mohammad Omidvar Tehrani, Alvyn Kang, Arnell Kang, Zeyu Chen, Ang\'elica Moreira, Xuan Feng, Angel X. Chang, Nick Sumner, Steven Y. Ko
arXiv AI
Jul 7

Conditional Clifford-Steerable CNNs for PDE Modeling

arXiv:2510. 14007v2 Announce Type: replace-cross Abstract: We introduce Conditional Clifford-Steerable CNNs (C-CSCNNs), a unified framework that incorporates equivariance to arbitrary pseudo-Euclidean groups and significantly improves the expressivity of standard CSCNNs.

By B\'alint L\'aszl\'o Szarvas, Maksim Zhdanov
arXiv Machine Learning
Jul 7

HNSW with Accuracy Guarantees Using Graph Spanners

arXiv:2607. 02338v2 Announce Type: replace-cross Abstract: Hierarchical Navigable Small World (HNSW) graphs serve as the industry standard due to their logarithmic complexity and strong empirical performance.

By Minghao Li, Raghav Mittal, Sanjivni Rana, Suraj Shetiya, Gautam Das, Nick Koudas