Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv AI

Jun 9

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

arXiv:2606. 07591v1 Announce Type: cross Abstract: AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify.

By Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, Kun Li, Sheng Xu, Shengdu Chai, Fangchen Yu, Xiangyu Zhao, Zhangrui Zhao, Weijie Ma, Zijie Guo, Haoyu Zhou, Haoxiang Yin, Lixue Cheng, Chaofan Hu, Haoxuan Li, Lu Mi, Xuxuan Xie, Yifan Zhou, Ruizhe Chen, Zhiwang Zhou, Xingjian Guo, Yuhao Zhou, Xuming He, Shengyuan Xu, Xinyu Gu, Jiamin Wu, Mianxin Liu, Chunfeng Song, Fenghua Ling, Dongzhan Zhou, Shixiang Tang, Yuqiang Li, Mao Su, Peng Ye, Siqi Sun, Bin Wang, Xue Yang, Zhenfei Yin, Tianfan Fu, Guangtao Zhai, Wanli Ouyang, Bo Zhang, Lei Bai, Wenlong Zhang

llms agents multimodal benchmarks

arXiv AI

Jun 9

ComplexConstraints and Beyond: Expert Rubrics for RLVR

arXiv:2606. 09118v1 Announce Type: new Abstract: As LLM capabilities advance rapidly, the evaluation methods used to assess them increasingly lag behind.

By Sushant Mehta, Liudas Panavas, Edwin Chen

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 9

MeCo: One-Step MeanFlow-based Corrector for Multi-Channel Speech Separation

arXiv:2606. 09677v1 Announce Type: cross Abstract: While discriminative models for multi-channel speech separation excel in reference-based metrics, they often exhibit suboptimal human listening quality.

By Dohwan Kim, Jung-Woo Choi

benchmarks

arXiv Machine Learning

Jun 9

DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination

arXiv:2606. 08068v1 Announce Type: new Abstract: Multi-agent large language model (LLM) systems often fail to reliably outperform a single strong model equipped with best-of-N sampling.

By Yi Xie, Zhanke Zhou, Chentao Cao, Bo Liu, Bo Han

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jun 9

A Geometry-Aware Triplane Field Network for Vehicle Aerodynamic Prediction

arXiv:2606. 07724v1 Announce Type: new Abstract: High-fidelity computational fluid dynamics (CFD) is crucial to vehicle aerodynamic analysis, but its cost still constrains early-stage design exploration.

By Kangkang Qi, Huiyu Yang, Keqi Ding, Yunpeng Wang, Yuntian Chen, Yuanwei Bin, Rikui Zhang, Jianchun Wang

benchmarks

arXiv AI

Jun 9

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

arXiv:2606. 09426v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools.

By Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li, Caihua Shan

agents benchmarks

arXiv Machine Learning

Jun 9

KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

arXiv:2606. 07651v1 Announce Type: new Abstract: Traditional fake news detection methods are falling behind as multimodal misinformation grows more advanced, seamlessly blending deceptive text, manipulated visuals, and factually incorrect claims.

By Kevin Patel, Shashi Bhushan Jha

llms multimodal benchmarks safety

arXiv AI

Jun 9

A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

arXiv:2606. 07718v1 Announce Type: new Abstract: Agentic AI tools offer a promising path to automating software development bottlenecks in scientific research pipelines, particularly for stages that take domain experts days to months to build, where scientists care about correctness and robustness, not implementation details.

By Kai A. Horstmann, Ethan Lin, Alice A. Robie, Jennifer J. Sun, Kristin Branson

agents benchmarks

arXiv AI

Jun 9

Towards Long-Horizon Vessel Trajectory and Destination Forecasting with Reasoning Large Language Models

arXiv:2606. 08633v1 Announce Type: new Abstract: Long-horizon maritime trajectory prediction is important for shipping management, logistics planning, and maritime risk analysis, yet month-level forecasting remains insufficiently studied.

By Hongwei Wang, Miao Zhou, Fengde Wang, Yuting Wang, Jiewen Yu, Jun-Yan He, Bohao Qu, Wanbing Zhang, Xiuju Fu, Qing Guo, Zipei Fan, Yingying Xing, Yi Yuan

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 9

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

arXiv:2606. 09131v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens.

By Siyuan Liu, Jinyang Wu

llms diffusion multimodal benchmarks

arXiv Machine Learning

Jun 9

Position: Genomic Model Research Must Move Beyond Anecdotal Evaluation of Interpretability Methods

arXiv:2606. 07607v1 Announce Type: new Abstract: Advances in machine learning and computational power have unlocked the predictive potential of the human genome, yet biologists now demand that these models also elucidate the underlying biological mechanisms.

By Shasha Zhou, Mingyu Huang, Ke Li

benchmarks safety

arXiv AI

Jun 9

HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions

arXiv:2503. 14229v4 Announce Type: replace Abstract: Vision-and-Language Navigation (VLN) has been studied mainly in either discrete or continuous spaces, with little attention to dynamic, crowded environments.

By Yifei Dong, Fengyi Wu, Qi He, Lingdong Kong, Heng Li, Minghan Li, Zebang Cheng, Yuxuan Zhou, Jingdong Sun, Qi Dai, Alexander G Hauptmann, Zhi-Qi Cheng

agents robotics benchmarks safety

arXiv AI

Jun 9

Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture

arXiv:2606. 09500v1 Announce Type: new Abstract: Objective.

By Yoojin Nam, Jinhoon Jeong, Namkug Kim

llms benchmarks

arXiv Machine Learning

Jun 9

Decomposable Neuro Symbolic Regression

arXiv:2511. 04124v3 Announce Type: replace Abstract: Symbolic regression (SR) models complex systems by discovering mathematical expressions that capture underlying relationships in observed data.

By Giorgio Morales, John W. Sheppard

llms benchmarks

arXiv AI

Jun 9

MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance

arXiv:2605. 22664v2 Announce Type: replace Abstract: LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions.

By Thomson Yen, Julian Poeltl, Harshith Srinivas Gear, Yilin Meng, Joshua Fan, Adam Shen, Yili Liu, Ali Bauyrzhan, Siri Du, Haoyang Liu, Daniel Guetta, Hongseok Namkoong

llms agents benchmarks

arXiv AI

Jun 9

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

arXiv:2606. 09669v1 Announce Type: new Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world.

By Hongcheng Gao, Hailong Qu, Jingyi Tang, Jiahao Wang, Zihao Huang, Hengkang Qiao, Shihong Huang, Junming Yang, Yi Li, Hongyixuan Yuan, Wenjie Li, Bohan Zeng, Wenbo Li, Bo Wang, Jianhui Liu, Olive Huang, Haoyang Huang, Wentao Zhang, Guoqing Huang, Nan Duan, Yinpeng Dong

llms agents multimodal benchmarks

arXiv AI

Jun 9

Bayesian Selective Latent Inference for Wastewater-First Influenza Monitoring

arXiv:2606. 09433v1 Announce Type: new Abstract: Wastewater influenza surveillance can reveal community circulation before clinical reporting, but wastewater alone is not a fully identifiable proxy for human burden.

By Yixuan Zhang (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark), Yang Song (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark), Hao Wang (Rutgers University, New Brunswick, NJ, USA), Samir Bhatt (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark, MRC Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Faculty of Medicine, Imperial College London, London, United Kingdom), Hengguan Huang (Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark, MRC Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Faculty of Medicine, Imperial College London, London, United Kingdom)

benchmarks

arXiv AI

Jun 9

TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models

arXiv:2510. 27544v2 Announce Type: replace Abstract: Temporal reasoning involves understanding how systems evolve over time through input-driven state transitions.

By Nikolaus Holzer, William Fishell, Baishakhi Ray, Mark Santolucito

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 9

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

arXiv:2606. 08611v1 Announce Type: cross Abstract: We study data-driven real-time economic optimization of a multi-product chemical reactor when no reliable first-principles model is available beyond a steady-state energy balance.

By Liqiu Dong, Marta Zag\'orowska, Mehmet Mercang\"oz

benchmarks

arXiv Machine Learning

Jun 9

Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control

arXiv:2512. 01467v2 Announce Type: replace Abstract: Controlling autonomous systems under real-world conditions often requires policies that can be evaluated with low latency and minimal energy consumption.

By Fabian Kresse, Christoph H. Lampert

agents efficiency benchmarks

Get the daily briefing