Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 30

UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration

arXiv:2604. 16325v3 Announce Type: replace-cross Abstract: Multivariate time series forecasting is fundamental to numerous domains such as energy, finance, and environmental monitoring, where complex temporal dependencies and cross-variable interactions pose enduring challenges.

By Xingsheng Chen, Xianpei Mu, Deyu Yi, Yilin Yuan, Xingwei He, Bo Gao, Regina Zhang, Pietro Lio, Siu-Ming Yiu

llms benchmarks

arXiv AI

Jun 30

Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon

arXiv:2605. 09708v2 Announce Type: replace-cross Abstract: We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in $n$-body problems, multi-field Boltzmann, neighbor-list molecular dynamics, multi-kernel PDE, FFT).

By V\'ictor Gallego

llms agents benchmarks

arXiv Machine Learning

Jun 30

The Contagion Tensor: A Framework for Measuring Output-Distribution Coupling in Multi-Agent LLM Systems -- and Auditing the Claims It Enables

arXiv:2606. 28839v1 Announce Type: new Abstract: We introduce the Contagion Tensor, a measurement framework for quantifying how large language model (LLM) output distributions couple across modalities, agents, and time steps.

By Zewen Liu

llms agents benchmarks

arXiv Machine Learning

Jun 30

Active Quantum Kernel Acquisition for Gaussian Process Regression

arXiv:2606. 28833v1 Announce Type: new Abstract: Quantum kernel estimation on near-term hardware is shot-budgeted: every entry of the kernel Gram matrix is a Bernoulli expectation that must be sampled with a finite number of circuit executions.

By Jian Xu, Delu Zeng, Qibin Zhao

benchmarks

arXiv Machine Learning

Jun 30

ML-Powered LDAP Reconnaissance Detection using Weak Supervision

arXiv:2606. 28917v1 Announce Type: new Abstract: Lightweight Directory Access Protocol (LDAP) is a protocol that allows users to query and modify Active Directory (AD) data.

By Shaefer Drew, Edward Raff, Michael Brautbar, Yaron Zinar, Benjamin Malmberg, Dor Agron, Sagi Sheinfeld, Avraham Kama, Asaf Romano

benchmarks

arXiv Machine Learning

Jun 30

STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

arXiv:2606. 29592v1 Announce Type: new Abstract: A central premise of autonomous scientific imaging is that smarter navigation, whether Bayesian, RL-based, or otherwise adaptive, is the principal lever for sample-efficient acquisition.

By Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 30

A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

arXiv:2606. 29719v1 Announce Type: new Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it.

By Liu Zewen

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 30

GLIP: Graph and LLM Joint Pretraining for Graph-Level Tasks

arXiv:2606. 29773v1 Announce Type: new Abstract: Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine.

By Haoxin Sun, Yiqing Lin, Yajun Huang, Chenhui Dong, Mingjun Li, Zhongzhi Zhang

llms diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation

arXiv:2606. 30059v1 Announce Type: new Abstract: Industry-scale video and live-streaming moderation imposes requirements that are difficult to satisfy with generic pretrained public models or external APIs, including adaptation to platform-specific data distributions, policy-specific objectives, and product-level safety constraints.

By Shuchang Ye, Jinqiang Yu, Zhujun Xiao, Yajing Kong, Yist Y. Lin, Yang Ma, Jiaxi Liu, Xiaolei Xu, Zheng Yu

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Toward an Energy-Optimized Operation of Data Centers Located in Wind Farms Using Reinforcement Learning

arXiv:2606. 30316v1 Announce Type: new Abstract: This paper studies Reinforcement Learning as an online controller for curtailment-aware workload shifting in wind-turbine-integrated high-performance computing (HPC) data centers.

By Jan Stenner, Alexander Kilian, Sebastian Peitz, Hermann de Meer

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.

By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He

llms agents benchmarks

arXiv AI

Jun 30

Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization

arXiv:2602. 11351v2 Announce Type: replace Abstract: Proactive large language model (LLM) agents aim to actively plan, query, and interact over multiple turns, enabling efficient task completion beyond passive instruction following and making them essential for real-world, user-centric applications.

By Yihang Yao, Zhepeng Cen, Haohong Lin, Shiqi Liu, Zuxin Liu, Jiacheng Zhu, Zhang-Wei Hong, Laixi Shi, Ding Zhao

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

Shoot from the HIP: Hessian Interatomic Potentials without derivatives

arXiv:2509. 21624v3 Announce Type: replace Abstract: Fundamental tasks in computational chemistry, from transition state search to vibrational analysis, rely on molecular Hessians, which are the second derivatives of the potential energy.

By Andreas Burger, Luca Thiede, Nikolaj R{\o}nne, Varinia Bernales, Nandita Vijaykumar, Tejs Vegge, Arghya Bhowmik, Alan Aspuru-Guzik

benchmarks

arXiv Machine Learning

Jun 30

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

arXiv:2606. 30319v1 Announce Type: cross Abstract: Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience.

By Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng, Mianxin Liu, Chi Zhang, Wanli Ouyang, Chunfeng Song, Changqing Zhang, Jiamin Wu

nlp multimodal benchmarks safety

arXiv AI

Jun 30

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

arXiv:2601. 03555v3 Announce Type: replace Abstract: Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning.

By Yuxuan Jiang, Francis Ferraro

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

Bridging the NISQ and Fault-Tolerant Regimes: Generative-ML-Assisted Quantum Selected CI for Molecular Simulations

arXiv:2606. 30551v1 Announce Type: cross Abstract: Calculation of binding energies for protein-ligand molecular systems requires accurate treatment of the electronic structure, a quantum chemistry problem that scales exponentially on classical hardware, while current quantum hardware remains too noisy for the required circuit depths.

By Anurag K. S. V., Ashish Kumar Patra, Manas Mukherjee, Ruchika Bhat, Sai Shankar P., Rahul Maitra, Jaiganesh G

benchmarks

arXiv Machine Learning

Jun 30

$\mu$Flow: Leveraging Average Images for Improving Generalisation of Deepfake Faces Detectors

arXiv:2606. 30528v1 Announce Type: cross Abstract: Current generative models, including GANs and diffusion models, have reached an outstanding level of photorealism, posing significant risks to privacy and security.

By Orazio Pontorno, Mattia Litrico, Luca Guarnera, Mario Valerio Giuffrida, Sebastiano Battiato

diffusion benchmarks safety

arXiv Machine Learning

Jun 30

Friend or Foe

arXiv:2509. 00123v2 Announce Type: replace-cross Abstract: A fundamental challenge in microbial ecology is determining whether bacteria compete or cooperate in different environmental conditions.

By Oleksandr Cherednichenko, Josephine Solowiej-Wedderburn, Laura M. Carroll, Eric Libby

benchmarks

arXiv Machine Learning

Jun 30

CoLR-Det: Collaborative Latent Restoration for Small Object Detection in Low-Resolution Remote Sensing Images

arXiv:2601. 12507v2 Announce Type: replace-cross Abstract: Low-resolution remote sensing small object detection is limited by both missing visual details and the ambiguity of how details serve detection.

By Ruo Qi, Linhui Dai, Yusong Qin, Chaolei Yang, Yanshan Li

computer-vision benchmarks

arXiv AI

Jun 30

Database Context Compression for Text-to-SQL on Real-World Large Databases

arXiv:2606. 28601v1 Announce Type: cross Abstract: Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks such as Spider 2.

By Jingwen Liu, Weibin Liao, Xin Gao, Junfeng Zhao, Yasha Wang

llms benchmarks data-engineering

Get the daily briefing