Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv Machine Learning

Jul 22

Finite-Agent Stochastic Differential Games on Large Graphs: II. Graph-Based Architectures

arXiv:2509. 12484v2 Announce Type: replace Abstract: We propose a novel neural network architecture, called Non-Trainable Modification (NTM), for computing Nash equilibria in stochastic differential games (SDGs) on graphs.

By Ruimeng Hu, Jihao Long, Haosheng Zhou

rag agents robotics benchmarks safety

arXiv AI

Jul 22

ConceptCF: Concept-based Counterfactuals for the Explainability of Time Series

arXiv:2607. 18748v1 Announce Type: cross Abstract: This paper proposes ConceptCF, a method for counterfactual generation that operates on human-interpretable concepts.

By Annemarie Jutte, Faizan Ahmed, Jeroen Linssen, Maurice van Keulen

efficiency benchmarks safety

arXiv AI

Jul 22

Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR

arXiv:2607. 18254v1 Announce Type: new Abstract: Multi-Level Intermediate Representation (MLIR) underlies modern ML compiler infrastructure (TensorFlow, JAX/StableHLO, PyTorch Inductor, IREE), yet appears only in trace amounts in code-LM pretraining corpora.

By Plawan Kumar Rath

fine-tuning benchmarks

arXiv Machine Learning

Jul 22

Neural Kolmogorov Equations: Parallelizable Learning of Stochastic Dynamics under General Noise

arXiv:2607. 19173v1 Announce Type: new Abstract: Neural stochastic differential equations (SDEs) have emerged as powerful tools for learning noisy or stochastic dynamics directly from data; however, existing approaches largely assume uncoupled and continuous noise, limiting their applicability to realistic stochastic drivers, and often scale poorly in time, requiring expensive autoregressive training.

By Arthur Bizzi, Olga Fink

benchmarks

arXiv Machine Learning

Jul 22

Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Robotics

arXiv:2607. 18540v1 Announce Type: cross Abstract: Robotic perception pipelines increasingly rely on large vision backbones deployed on SWaP-constrained edge platforms, making post-training quantization (PTQ) attractive for real-time inference.

By Hamidreza Yaghoubi Araghi, Parastoo Pilevar, Ming C. Lin

llms robotics fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 22

Weak-to-Strong Learning in Decision Making

arXiv:2607. 18467v1 Announce Type: new Abstract: Many operational decisions rely on predictive models that estimate uncertain outcomes conditional on observable contexts.

By Jingwei Ji, Renyuan Xu

benchmarks

arXiv Machine Learning

Jul 22

Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning

arXiv:2512. 18763v2 Announce Type: replace Abstract: Unlike their conventional use as estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses.

By Minh Vu, Konstantinos Slavakis

reinforcement-learning benchmarks

arXiv AI

Jul 22

Robust Reasoning Benchmark

arXiv:2604. 08571v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting.

By Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey

llms nlp benchmarks

arXiv Machine Learning

Jul 22

Robust Multi-View Classification under Noisy Supervision via Global Anchor Consensus

arXiv:2607. 18561v1 Announce Type: new Abstract: In recent years, multi-view learning has attracted increasing attention, as it integrates the complementary information of heterogeneous views.

By Yuliang Yang, Hongzhe Zhang, Huiru Wang

benchmarks

arXiv AI

Jul 22

BioSecBench-Surveillance: A Verifiable Benchmark for AI Agents in Pathogen Genomic Surveillance

arXiv:2607. 19262v1 Announce Type: new Abstract: As pathogen genomic surveillance scales, the bottleneck is shifting from data generation to analysis.

By Harmon Bhasin, Kevin Flyangolts, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Amanda Darling, Joshua Stallings, David Stern, Shawn Higdon, Claire Duvallet, Bryan Tegomoh, Kenny Workman

llms agents benchmarks

arXiv AI

Jul 22

Now We Know? A Systematic Comparison of TerraMind and THOR

arXiv:2607. 18504v1 Announce Type: cross Abstract: Benchmarks for Geospatial Foundation Models (GFMs) increasingly rank models by aggregate score, but such rankings obscure why models differ: how much of the gap is architecture, how much is decoder capacity, and how much is a use-case-specific artefact?

By Frederick Schindlegger, Kenzo Bounegta, Eva Gmelich Meijling, Johannes Jakubik, Arnt-B{\o}rre Salberg, Theodor Forgaard, Nicolas Longepe, Valerio Marsocci

computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 22

Chi-Square Wavelet Graph Neural Networks for Heterogeneous Graph Anomaly Detection

arXiv:2505. 18934v2 Announce Type: replace-cross Abstract: Graph Anomaly Detection (GAD) in heterogeneous networks presents unique challenges due to node and edge heterogeneity.

By Xiping Li, Xiangyu Dong, Xingyi Zhang, Kun Xie, Yuanhao Feng, Bo Wang, Guilin Li, Wuxiong Zeng, Xiujun Shu, Sibo Wang

benchmarks safety

arXiv AI

Jul 22

TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

arXiv:2506. 18421v3 Announce Type: replace-cross Abstract: The majority of data in businesses and industries is stored in tables, databases, and data warehouses.

By Ce Li, Xiaofan Liu, Zhiyan Song, Ce Chi, Boshen Shi, Chen Zhao, Guanguang Chang, Zhendong Wang, Kexin Yang, Xing Wang, Chao Deng, Junlan Feng

llms benchmarks

arXiv AI

Jul 22

Semantic Cooperative Games for Contribution Attribution in LLM-Based Multi-Agent Systems

arXiv:2607. 18255v1 Announce Type: new Abstract: Contribution attribution has become a central problem in LLM-based multi-agent systems, where final outputs are produced through multiple agents, message exchanges, and ordered workflow dependencies.

By Pengyi Jiang, Xiaoguang Zhu, Quanyan Zhu

llms agents benchmarks

arXiv AI

Jul 22

SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

arXiv:2607. 18960v1 Announce Type: cross Abstract: Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring.

By Arther Tian, Alex Ding, Simon Wu, Aaron Chan

llms fine-tuning benchmarks

arXiv AI

Jul 22

Data Leakage Prevention in Agentic Applications via Preemptive Hardening

arXiv:2607. 18847v1 Announce Type: cross Abstract: Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks.

By Akansha Shukla, Emily Bellov, Parth Atulbhai Gandhi, Yuval Elovici, Asaf Shabtai

llms agents robotics benchmarks safety

arXiv Machine Learning

Jul 22

Enhanced NQS via Annealed Gradient Descent

arXiv:2607. 18865v1 Announce Type: cross Abstract: Neural quantum states offer expressive representations of quantum many-body wave functions, yet their practical accuracy can be limited by stochastic optimization rather than representational capacity.

By Shiwei Zhou, Yiming Huang, Xiao Yuan, Xiaoxia Cai

benchmarks

arXiv AI

Jul 22

Domain Design for the Cops and Robbers Problem

arXiv:2607. 18274v1 Announce Type: cross Abstract: Cops and Robbers is a well-studied problem in graph theory.

By Connor Little, Meagan Mann, Erin Meger, Christian Muise

benchmarks

arXiv AI

Jul 22

When Does Machine Learning Beat Value Sorting? A Three-Dataset Diagnostic of Exposure-Weighted Shipment Prioritization

arXiv:2607. 18573v1 Announce Type: new Abstract: Delay-risk models are usually judged by predictive accuracy.

By Jize Li

benchmarks safety

arXiv Machine Learning

Jul 22

Quantum Reservoir Computing: Recent Advances and Future Directions

arXiv:2607. 18552v1 Announce Type: cross Abstract: Quantum reservoir computing (QRC) uses the dynamics of a fixed or weakly tuned quantum system to transform temporal and sequential inputs into measured features, while training is typically confined to a classical readout.

By Shehbaz Tariq, Muhammad Talha, Arshid Ali, Muhammad Diyan, Symeon Chatzinotas

benchmarks

Get the daily briefing