Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv AI

Jul 8

Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS)

arXiv:2607. 06505v1 Announce Type: cross Abstract: GitHub hosts hundreds of millions of public repositories, but the platform exposes no native mapping from repositories to standardized industry sectors.

By Kevin Xu, Alexander Quispe

llms rag diffusion fine-tuning benchmarks

arXiv AI

Jul 8

Driving the Wrong Way: Leveraging Interpretability in End2End Autonomous Driving Models

arXiv:2607. 06328v1 Announce Type: new Abstract: The increasing adoption of end-to-end learning for autonomous driving introduces increased model complexity and opacity, raising the risk of learning undesired or erroneous behavior.

By Franz Motzkus, Sebastian Bernhard

agents robotics benchmarks safety

arXiv AI

Jul 8

CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries

arXiv:2607. 05405v1 Announce Type: cross Abstract: To interact with users fairly and without stereotyping, AI models must display cultural competency, i.

By Vasudha Varadarajan, Akhila Yerukola, Mona T. Diab, Maarten Sap

llms benchmarks

arXiv Machine Learning

Jul 8

Classification of Financial Data Using Quantum Support Vector Machine

arXiv:2412. 10860v2 Announce Type: replace-cross Abstract: Quantum Support Vector Machine is a kernel-based approach to classification problems.

By Seemanta Bhattacharjee, MD. Muhtasim Fuad, A. K. M. Fakhrul Hossain

benchmarks

arXiv AI

Jul 8

Replication in Visual Diffusion Models: A Survey and Outlook

arXiv:2408. 00001v2 Announce Type: replace-cross Abstract: Visual diffusion models have revolutionized the field of creative AI, producing high-quality and diverse content.

By Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, Yi Yang

diffusion benchmarks safety

arXiv AI

Jul 8

ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation

arXiv:2607. 05750v1 Announce Type: new Abstract: Computer-aided design (CAD) for industrial components requires long-horizon procedural modeling, robust feature dependencies, editable parametric geometry, and production-grade B-Rep execution.

By Yunhan Xu, Qifeng Wu, Xunjin Li, Yuanwei Bin, Qingsong Yao, Jianghang Gu, Guan Wang, Weihao Lv, Huiyu Yang, Wenfa Luo, Jiao Xiang, Yuntian Chen, Shiyi Chen

agents efficiency benchmarks

arXiv Machine Learning

Jul 8

Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

arXiv:2604. 04738v2 Announce Type: replace-cross Abstract: Fine-tuning is the dominant paradigm for adapting large machine learning models, yet current deployment pipelines provide no way to verify how a released model was updated.

By Zhenhang Shang, Yingzhe Yu, Kani Chen

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 8

A semantic mutation metric for metamorphic relation adequacy in scientific computing programs

arXiv:2605. 17437v2 Announce Type: replace-cross Abstract: Context.

By Meng Li (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Xiaohua Yang (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Jie Liu (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Shiyu Yan (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China)

llms benchmarks

arXiv Machine Learning

Jul 8

Data-Efficient Indentation Size Effect Correction in Steels Using Machine Learning and Physics-Constrained Neural Network

arXiv:2604. 27775v2 Announce Type: replace-cross Abstract: Shallow nanoindentation enables mechanical characterization of thin films, individual phases, and other volume-constrained materials, but the measured hardness is inflated by the indentation size effect (ISE).

By Radmir Karamov, Tagir Karamov

benchmarks

arXiv Machine Learning

Jul 8

Joint Energy Management and Coordinated AIGC Workload Scheduling for Distributed Data Centers: A Diffusion-Aided Reward Shaping Approach

arXiv:2605. 02965v2 Announce Type: replace Abstract: Artificial intelligence-generated content (AIGC) has emerged as a transformative paradigm for automating the creation of diverse and customized content, giving rise to rapidly growing computational workloads in cloud data centers.

By Yang Fu, Peng Qin, Liming Chen, Zihao Zhang, Hao Yu, Yifei Wang

diffusion reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 8

Leech Lattice Vector Quantization for Efficient LLM Compression

arXiv:2603. 11021v2 Announce Type: replace Abstract: Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds.

By Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough, Markus Nagel

llms efficiency benchmarks

arXiv Machine Learning

Jul 8

GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

arXiv:2601. 20753v4 Announce Type: replace Abstract: Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) approximates diverse Pareto-optimal solutions by conditioning a single policy on user-specified preferences, enabling run-time adaptation to arbitrary trade-offs without retraining.

By Zhiheng Jiang, Yunzhe Wang, Ryan Marr, Ellen Novoseller, Benjamin T. Files, Volkan Ustun

reinforcement-learning benchmarks

arXiv Machine Learning

Jul 8

Exogenous Dropout: A Simple, Strong Baseline for Corruption-Robust Time Series Forecasting with Covariates

arXiv:2607. 05452v1 Announce Type: new Abstract: Time series forecasters that use exogenous covariates are fragile in deployment: when those covariates are noised, temporally misaligned, or missing, strong exogenous-fusion and exogenous-adapted models can degrade far above the endogenous-only floor.

By Hao Hu, Xue-shan Ai

benchmarks

arXiv AI

Jul 8

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

arXiv:2605. 30794v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks.

By Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing

llms nlp multimodal benchmarks

arXiv AI

Jul 8

MambaGaze: Bidirectional Mamba with Explicit Missing Data Modeling for Cognitive Load Assessment from Eye-Gaze Tracking Data

arXiv:2605. 22775v2 Announce Type: replace-cross Abstract: Real-time cognitive load assessment from eye-tracking signals could enable adaptive human-centered AI in safety-critical applications such as driver vigilance monitoring or automated flight deck assistance, yet two challenges persist: handling frequent data missingness from blinks and tracking failures, and efficiently modeling long-range temporal dependencies.

By Amir Mousavi, Mohammad Sadegh Sirjani, Erfan Nourbakhsh, Mimi Xie, Rocky Slavin, Leslie Neely, John Davis, John Quarles

efficiency benchmarks safety

arXiv AI

Jul 8

Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks

arXiv:2604. 23931v2 Announce Type: replace-cross Abstract: Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data.

By Chi-Sheng Chen, En-Jui Kuo

llms benchmarks

arXiv AI

Jul 8

StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systems

arXiv:2607. 05844v1 Announce Type: new Abstract: Agent systems accumulate conflicting observations across branches, retries, and replicas, yet many practical memory layers still collapse disagreement behind overwrite rules that are difficult to inspect or correct.

By Sergey Volkov, Yang Li, Ye Luo

agents benchmarks

arXiv AI

Jul 8

Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure

arXiv:2607. 05805v1 Announce Type: new Abstract: Dilution refrigerators are the enabling infrastructure of superconducting quantum computers, yet their fault diagnosis is still dominated by threshold alarms that report that something is wrong, not what.

By Praneeth Narisetty, Uday Kumar Reddy Kattamanchi, Shiva Nagendra Babu Kore

llms agents robotics benchmarks

arXiv AI

Jul 8

x-Prediction Is All You Need:Training-Free Accelerated Generation via Endpoint Decodability

arXiv:2607. 06114v1 Announce Type: cross Abstract: Diffusion and flow matching models generate high-quality samples, but their ODE samplers often need tens to hundreds of neural function evaluations (NFEs).

By Xin Peng, Ang Gao

diffusion efficiency benchmarks

arXiv AI

Jul 8

PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation

arXiv:2607. 05915v1 Announce Type: new Abstract: PCB routing is the task of connecting the nets of a board with copper traces under strict design rules, yet learning-based methods still lag behind rule-based routers.

By Hyungseok Song, Junseok Park, Won-Seok Choi, Seohui Bae, Han-Seul Jeong, Youngjoon Park, Soonyoung Lee

llms agents reinforcement-learning benchmarks

Get the daily briefing