Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv AI

Jul 8

Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure

arXiv:2607. 05805v1 Announce Type: new Abstract: Dilution refrigerators are the enabling infrastructure of superconducting quantum computers, yet their fault diagnosis is still dominated by threshold alarms that report that something is wrong, not what.

By Praneeth Narisetty, Uday Kumar Reddy Kattamanchi, Shiva Nagendra Babu Kore

llms agents robotics benchmarks

arXiv AI

Jul 8

StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systems

arXiv:2607. 05844v1 Announce Type: new Abstract: Agent systems accumulate conflicting observations across branches, retries, and replicas, yet many practical memory layers still collapse disagreement behind overwrite rules that are difficult to inspect or correct.

By Sergey Volkov, Yang Li, Ye Luo

agents benchmarks

arXiv Machine Learning

Jul 8

Data-Efficient Indentation Size Effect Correction in Steels Using Machine Learning and Physics-Constrained Neural Network

arXiv:2604. 27775v2 Announce Type: replace-cross Abstract: Shallow nanoindentation enables mechanical characterization of thin films, individual phases, and other volume-constrained materials, but the measured hardness is inflated by the indentation size effect (ISE).

By Radmir Karamov, Tagir Karamov

benchmarks

arXiv Machine Learning

Jul 8

A semantic mutation metric for metamorphic relation adequacy in scientific computing programs

arXiv:2605. 17437v2 Announce Type: replace-cross Abstract: Context.

By Meng Li (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Xiaohua Yang (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Jie Liu (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Shiyu Yan (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China)

llms benchmarks

arXiv AI

Jul 8

RMISC: A Large-scale Real-world Multivariate Corpus for Time Series Foundation Models

arXiv:2607. 06504v1 Announce Type: new Abstract: Recent years have witnessed the emergence of multivariate modeling using time series foundation models (TSFMs), which achieve advanced zero-shot generalization.

By Qian Sun, Yong-Ming Tian, Jia-Wei Huang, Cheng Feng, Shao-Qun Zhang

benchmarks

arXiv AI

Jul 8

Universal Algorithm-Implicit Learning

arXiv:2602. 14761v2 Announce Type: replace-cross Abstract: Current meta-learning methods are constrained to narrow task distributions with fixed feature and label spaces, limiting applicability.

By Stefano Woerner, Seong Joon Oh, Christian F. Baumgartner

llms rag nlp multimodal benchmarks

arXiv AI

Jul 8

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.

By Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesus Olivera

llms nlp efficiency benchmarks

arXiv AI

Jul 8

EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems

arXiv:2607. 05638v1 Announce Type: cross Abstract: Teams deploying large language models in business contexts need evaluation systems, yet most treat evaluation as static model selection: run benchmarks, rank models, deploy the winner.

By Kenneth Benavides, Josh Fleischer, Danti Chen

llms agents benchmarks safety

arXiv AI

Jul 8

From Global to Granular: Revealing IQA Model Performance via Correlation Surface

arXiv:2601. 21738v2 Announce Type: replace-cross Abstract: Evaluation of Image Quality Assessment (IQA) models has long been dominated by global correlation metrics, such as Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-Order Correlation Coefficient (SRCC).

By Baoliang Chen, Danni Huang, Hanwei Zhu, Lingyu Zhu, Wei Zhou, Shiqi Wang, Yuming Fang, Weisi Lin

benchmarks

arXiv AI

Jul 8

From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation

arXiv:2603. 15600v2 Announce Type: replace-cross Abstract: Accurate process supervision remains a critical challenge for long-horizon robotic manipulation.

By Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu

reinforcement-learning robotics fine-tuning benchmarks

arXiv Machine Learning

Jul 8

Federated Physics-Grounded Reinforcement Learning for Distributed Stability Control in Smart Grids

arXiv:2607. 05553v1 Announce Type: new Abstract: Transient stability control in smart grids requires rapid post-fault damping of generator frequency and rotor angle deviations to prevent cascading failures.

By Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat, Eman Hammad

agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

arXiv:2604. 08435v2 Announce Type: replace-cross Abstract: It remains challenging to assess driver fatigue from untrimmed videos under constrained computational budgets, due to the difficulty of modeling long-range temporal dependencies in subtle facial expressions.

By Changdao Chen, Qinqiuhong Ye, Hao Chen, Jinyu Wang

efficiency multimodal benchmarks

arXiv Machine Learning

Jul 8

Exogenous Dropout: A Simple, Strong Baseline for Corruption-Robust Time Series Forecasting with Covariates

arXiv:2607. 05452v1 Announce Type: new Abstract: Time series forecasters that use exogenous covariates are fragile in deployment: when those covariates are noised, temporally misaligned, or missing, strong exogenous-fusion and exogenous-adapted models can degrade far above the endogenous-only floor.

By Hao Hu, Xue-shan Ai

benchmarks

arXiv Machine Learning

Jul 8

Classification of Financial Data Using Quantum Support Vector Machine

arXiv:2412. 10860v2 Announce Type: replace-cross Abstract: Quantum Support Vector Machine is a kernel-based approach to classification problems.

By Seemanta Bhattacharjee, MD. Muhtasim Fuad, A. K. M. Fakhrul Hossain

benchmarks

arXiv Machine Learning

Jul 8

Fine-Tuning Regimes Define Distinct Continual Learning Problems

arXiv:2604. 21927v3 Announce Type: replace Abstract: Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge.

By Paul-Tiberiu Iordache, Elena Burceanu

fine-tuning benchmarks

arXiv Machine Learning

Jul 8

Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

arXiv:2604. 04738v2 Announce Type: replace-cross Abstract: Fine-tuning is the dominant paradigm for adapting large machine learning models, yet current deployment pipelines provide no way to verify how a released model was updated.

By Zhenhang Shang, Yingzhe Yu, Kani Chen

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 8

PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.

By Daryna Dementieva, Nikolay Babakov, Kathy H\"ammerl, Ilseyar Alimova, Jind\v{r}ich Libovick\'y, Shu Okabe, Miras Baisbay, Lukas Edman, Abrorkhon Inomkhujaev, Antonia Karamolegkou, Mateusz Lango, Volkan \"Ozer, Nikola Selic, Subhankar Swain, Tsedeniya Kinfe Temesgen, Galit Bary Weisberg, Alexander Fraser

llms benchmarks

arXiv AI

Jul 8

OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

arXiv:2512. 23020v3 Announce Type: replace-cross Abstract: 3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes.

By Wenyuan Huang, Zhenyu Zhang, Zhao Wang, Zhou Wei, Ting Huang, Fang Zhao, Jian Yang

llms benchmarks

arXiv AI

Jul 8

FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games

arXiv:2607. 06514v1 Announce Type: new Abstract: We present FootsiesGym, an open-source environment for learning in a non-trivial two-player, zero-sum, imperfect-information game.

By Chase McDonald, Nathan Tsang, Wesley N. Kerr

reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

arXiv:2607. 05780v1 Announce Type: cross Abstract: While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization.

By Chuhao Zhou, Liquan Wang, Shuxin Cao, Xiangyu Chen, Yuxuan Hu, Boyu Ma, Animesh Garg, Jianfei Yang

robotics benchmarks

Get the daily briefing