Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 14

Diachronic Sample Integration: Robust Tail-Risk Estimation with Generative Models

arXiv:2607. 10810v1 Announce Type: cross Abstract: Deep generative models are increasingly used as simulators for downstream decision-making under data scarcity, but in risk-sensitive applications their usefulness depends on rare adverse scenarios rather than typical samples.

By Shuning Zhao, Patrick Wong, Leran Zhang, Xiaolin Hu

diffusion benchmarks safety

arXiv AI

Jul 14

BatteryLake: Agentic, Physics-Grounded Curation of Heterogeneous Battery Aging Data and Benchmarking

arXiv:2607. 09762v1 Announce Type: new Abstract: Public battery aging datasets are a critical asset for advanced health management, but their practical use is often limited by inconsistent formats, unclear schemas, and metadata scattered across repositories and publications.

By Tianwen Zhu, Hao Wang, Yonggang Wen

llms agents benchmarks

arXiv AI

Jul 14

Source-Lifted Flow Matching for Intervenable Multimodal Imitation

arXiv:2607. 10206v1 Announce Type: cross Abstract: Flow-matching policies are promising for imitation learning because they model complex multimodal action distributions.

By He Zhang, Ying Sun, Pengteng Li, Ziyang Chen, Yiren Zhao, Ziyang Rao, Weiyu Guo, Yandong Guo, Hui Xiong

diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jul 14

MeloBottleneck: Self-Supervised Melody Skeleton Extraction with a Latent Subsequence Bottleneck

arXiv:2607. 10233v1 Announce Type: cross Abstract: Melody skeleton extraction aims to derive a shorter melody that preserves structural notes while removing ornaments.

By Fan Bu, Rongfeng Li, Linfeng Fan

benchmarks safety

arXiv AI

Jul 14

A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents

arXiv:2605. 01143v2 Announce Type: replace Abstract: Large Language Model (LLM)-powered agents demonstrate strong capabilities in autonomous task execution, tool use, and multi-step reasoning.

By Sheldon Yu, Yingcheng Sun, Hanqing Guo, Qianqian Tong

llms agents benchmarks safety

arXiv Machine Learning

Jul 14

When Classical Baselines Are Tuned as Carefully as the Quantum Model, Does Quantum Reservoir Computing Still Win?

arXiv:2607. 09905v1 Announce Type: cross Abstract: Can a small quantum computer forecast a changing signal better than an ordinary classical method?

By Tushar Pandey

benchmarks

arXiv Machine Learning

Jul 14

SPORT: Structure-Aware Prototype Disentanglement for Incomplete Multi-View Clustering

arXiv:2607. 10413v1 Announce Type: cross Abstract: Prototype-based Incomplete Multi-view Clustering has recently attracted increasing attention by exploiting prototypes as semantic anchors for missing-view imputation.

By Yaoyuan Guo, Zhibin Gu, Songhe Feng, Yuhui Zheng, Bing Li

benchmarks

arXiv AI

Jul 14

Emotion Recognition in Signers

arXiv:2512. 15376v2 Announce Type: replace-cross Abstract: Recognition of signers' emotions suffers from one theoretical challenge and one practical challenge, namely, the overlap between grammatical and affective facial expressions and the scarcity of data for model training.

By Kotaro Funakoshi, Yaoxiong Zhu

llms benchmarks

arXiv Machine Learning

Jul 14

Toward Efficient Weakly Supervised Semantic Segmentation Using Only Low-Magnification Histopathological Images

arXiv:2607. 10783v1 Announce Type: cross Abstract: Whole-slide images (WSIs) provide rich tissue-level and cellular-level information, but storing and transmitting high-magnification pathology data is resource-intensive.

By Dung Minh Do, Nhat-Thanh Huynh, Duc Minh Huynh, Doanh C. Bui, Khang Nguyen

computer-vision benchmarks

arXiv Machine Learning

Jul 14

SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs

arXiv:2503. 08884v3 Announce Type: replace-cross Abstract: Unimodal vision models are known to rely on spurious correlations, but it remains unclear to what extent Multimodal Large Language Models (MLLMs) exhibit similar biases despite language supervision.

By Parsa Hosseini, Sumit Nawathe, Mazda Moayeri, Sriram Balasubramanian, Soheil Feizi

llms multimodal benchmarks safety

arXiv AI

Jul 14

QwenPaw-Data: Bridging Facts, Methodology, and Execution for Autonomous Enterprise Data Analytics

arXiv:2607. 11019v1 Announce Type: new Abstract: Enterprise data analysis is emerging as a distinct frontier for autonomous agents.

By Tianjing Zeng, Yuntao Hong, Zhongjun Ding, Dandan Liu, Yinan Mei, Yunxiang Su, Yiming Wang, Xiaojian Zhang, Jingyu Zhu, Junhao Zhu, Zhuowen Liang, Jiazhen Peng, Lianggui Weng, Zhihao Ding, Kerui Yi, Qifeng Wang, Rong Zhu, Bolin Ding, Liyu Mou, Jingren Zhou

agents benchmarks

arXiv AI

Jul 14

RVN-Bench: A Benchmark for Reactive Visual Navigation

arXiv:2603. 03953v2 Announce Type: replace-cross Abstract: Safe visual navigation is critical for indoor mobile robots operating in cluttered environments.

By Jaewon Lee, Jaeseok Heo, Gunmin Lee, Howoong Jun, Jeongwoo Oh, Songhwai Oh

agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 14

NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations

arXiv:2607. 10490v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents are attractive for network operations, but tickets, alerts, logs, runbooks, and ChatOps messages can carry indirect prompt injections.

By Ruksat Khan Shayoni, Muhammad Faraz Shoaib, S M Asif Hossain, M. F. Mridha

llms agents benchmarks safety

arXiv AI

Jul 14

MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

arXiv:2607. 11818v1 Announce Type: cross Abstract: We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.

By Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma, Afshin Dehghan

agents benchmarks

arXiv AI

Jul 14

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

arXiv:2607. 11871v1 Announce Type: cross Abstract: Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations.

By Zixiang Xu, Sixian Li, Huaxing Liu, Xiang Wang, Shuai Li, Zirui Song, Xiuying Chen

llms benchmarks safety

arXiv Machine Learning

Jul 14

AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating

arXiv:2607. 10593v1 Announce Type: new Abstract: Normalization is a critical component for stabilizing Transformer training, yet the choice between static strategies such as Layer Normalization (LN) and adaptive alternatives remains largely task-dependent.

By Piyush Kaushik Bhattacharyya, Divyanshu Rai, Swastik Singh, Kumar Aakash, Ayush Ranjan, Krutika Verma

llms nlp benchmarks

arXiv AI

Jul 14

AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking

arXiv:2607. 11526v1 Announce Type: cross Abstract: Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials.

By Hongxiao Li, Wanling Gao

benchmarks

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Jul 14

Adaptive Reinforcement Learning for Unobservable Random Delays

arXiv:2506. 14411v2 Announce Type: replace-cross Abstract: In standard reinforcement learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov decision process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately.

By John Wikman, Alexandre Proutiere, David Broman

agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 14

SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning

arXiv:2607. 11624v1 Announce Type: cross Abstract: Reinforcement learning (RL) algorithms classically suffer from poor sample efficiency.

By Evelyn D'Elia, Weishu Zhan, Giulio Turrisi, Giulio Romualdi, Giuseppe L'Erario, Raffaello Camoriano, Wei Pan, Daniele Pucci

agents reinforcement-learning robotics benchmarks

Get the daily briefing