Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 15

BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents

arXiv:2609.12394v2 Announce Type: replace Abstract: Mobile GUI agents are shifting from multi-module frameworks to native models trained end-to-end, yet industrial deployment faces three persistent g...

By Tong Ye, Kunyang Han, Guozhi Wang, Longqiang Luo, Zhifeng Ding, Yongxiang Zhang, Xiaolei Shen, Yuxuan Zhang, Zhuping Zhang, Tao Xu, Yue Pan, Yucheng Zhao, Yupei Hu, Yuanjiang Ouyang, Danfeng Shen, Runqi Lin, Hongda Cai, Zhaoxiong Wang, Mengjia Yan, Yingjie Zhong, Chen Zhou, Zeyu Zhang, Xuwen Zhu, Penggang Shi, Mingcheng Luo, Ziyang Wu, Min Jin, Mingfu Shen, Zairong Xu, Fan Zhang, Hao Wang, Liang Liu, Zhulin Xie, Lijun Yao, Xiao Liang, Liangmin Wen, Liqiang Feng, Feilong Wu, Min Hu, Min Chen, Guanjing Xiong, Xiaohu Ruan, Xiaoxin Chen

agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Sep 15

Understanding the Limits of Agentic ICD Coding

arXiv:2609.13806v1 Announce Type: cross Abstract: ICD-10-CM codes are alphanumeric codes used in the US to classify diagnoses and injuries for medical billing and epidemiological reporting. Standard...

By Chong Yock Eng, Yushi Cao, Yiming Chen, Kezhi Mao, Hongchao Jiang

agents benchmarks

arXiv AI

Sep 15

GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning

arXiv:2609.14066v1 Announce Type: cross Abstract: Although existing multi-agent Retrieval-Augmented Generation (RAG) systems have demonstrated promise on complex multimodal reasoning tasks, they rema...

By Zhongyu Wang

rag agents reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Sep 15

LIMBO: Lifelong Inference-Time Memory and Budget Optimization for LLM Agents

arXiv:2609.14138v1 Announce Type: cross Abstract: As LLM agents become integrated into increasingly complex workflows, they must continually acquire new capabilities while retaining competence on pre...

By Siddharth Sharma, Nilesh Prasad Pandey, Onat Gungor, Tajana Rosing

llms agents benchmarks

arXiv AI

Sep 15

Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarization

arXiv:2604.13175v2 Announce Type: replace-cross Abstract: Large language models can be aligned with human preferences through offline reinforcement learning (RL) on small labeled datasets. While sing...

By Aadyot Bhatnagar, Peter M{\o}rch Groth, Sebastian Ibarraran, Ali Madani

llms reinforcement-learning benchmarks safety

arXiv AI

Sep 15

Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

arXiv:2605.00022v2 Announce Type: replace-cross Abstract: The rapid proliferation of large audio models (LAMs) demands efficient approaches for model comparison, yet comprehensive benchmarks are cost...

By Woody Haosheng Gan, William Held, Diyi Yang

benchmarks safety

arXiv AI

Sep 15

Human-in-the-Loop Meta Bayesian Optimization for Fusion Energy and Scientific Applications

arXiv:2605.00068v2 Announce Type: replace-cross Abstract: Inertial Confinement Fusion (ICF) holds transformative promise for sustainable, near-limitless clean energy, yet remains constrained by prohi...

By Ricardo Luna Gutierrez, Sahand Ghorbanpour, Rahman Ejaz, Varchas Gopalaswamy, Riccardo Betti, Vineet Gundecha, Aarne Lees, Soumyendu Sarkar

benchmarks

arXiv AI

Sep 15

Compute Allocation for Self-Evolving LLMs: From Depth-Breadth to Multi-Armed Bandits

arXiv:2605.29268v3 Announce Type: replace-cross Abstract: LLM-guided evolutionary search (Evolve systems) has reached state-of-the-art results on mathematical and combinatorial tasks, yet most existi...

By Sixue Xing, Haoyu He, Kerui Wu, Zhuo Yang, Haozheng Luo, Tianfan Fu, Aarthy Nagarajan

llms reinforcement-learning benchmarks

arXiv AI

Sep 15

Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation

arXiv:2605.30984v2 Announce Type: replace-cross Abstract: Modern 3D medical vision-language models (VLMs) can generate fluent radiology-style text while exhibit critically low pathology detection and...

By Tom Maye-Lasserre, Yitong Li, Bailiang Jian, Morteza Ghahremani, Benedikt Wiestler, Christian Wachinger

llms multimodal benchmarks safety

arXiv Machine Learning

Sep 15

Machine Unlearning for Speech Question Answering in Large Audio-Language Models

arXiv:2609.13195v1 Announce Type: new Abstract: Large Audio-Language Models (LALMs) have recently shown strong capabilities in speech understanding and question answering (QA), but they also inherit...

By Zhe Liu

llms nlp fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Sep 15

When Compliance Data Masquerades as Evaluation: Measurement Validity for Deployed AI Systems

arXiv:2609.13642v1 Announce Type: new Abstract: We argue that a recurring failure in the evaluation of deployed AI systems occurs when data collected for operational monitoring or regulatory complian...

By Hung-Yu Lin, Xingran Huang, Qiming Guo, Jinwen Tang

benchmarks safety

arXiv Machine Learning

Sep 15

JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

arXiv:2609.13730v1 Announce Type: new Abstract: Reliable progress in offline policy learning depends on careful reporting, well-tuned baselines, and evaluation across diverse conditions. Prior work h...

By Nabil Omi, Eric Bae, Chung Yik Edward Yeung, Siddhartha Sen, Ali Farhadi

benchmarks

arXiv Machine Learning

Sep 15

Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulators

arXiv:2609.13819v1 Announce Type: new Abstract: Solving inverse problems with differentiable physics simulators holds the potential to revolutionize scientific discovery and engineering design, as it...

By Xiang Chen, Huanhuan Xia

benchmarks

arXiv Machine Learning

Sep 15

Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting

arXiv:2609.13840v1 Announce Type: new Abstract: A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters ar...

By Joo Ern Chin, Shih-Fen Cheng, Aldy Gunawan

benchmarks safety

arXiv Machine Learning

Sep 15

Machine Learning under Imperfect Data: Challenges and Methods

arXiv:2609.13914v1 Announce Type: new Abstract: Machine-learning models are commonly developed under an assumption that training and test data are sufficiently complete, balanced, labelled, and drawn...

By Masoumeh Zareapoor

benchmarks

arXiv Machine Learning

Sep 15

Optimal Transport for Efficient, Unsupervised Anomaly Detection on Industrial Data

arXiv:2609.13940v1 Announce Type: new Abstract: Effective anomaly detection frameworks are a central pillar of the Industry 4.0 paradigm. In this paper, we introduce an Optimal Transport (OT)-based f...

By Abigail Langbridge, Fearghal O'Donncha, James T Rayfield, Bradley Eck

benchmarks

arXiv Machine Learning

Sep 15

CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time

arXiv:2609.14239v1 Announce Type: new Abstract: Static benchmarks for computer-use agents fix a task set at release and score every system against it once. That makes them reproducible, and it lets t...

By Nitish Kovuru, Prateek Jannu

agents benchmarks

arXiv Machine Learning

Sep 15

Bayesian optimization with kernel ensembles and disagreement-based acquisition for source localization and acoustic inversion

arXiv:2609.14262v1 Announce Type: new Abstract: Joint source localization and geoacoustic inversion requires optimizing an objective built from an expensive normal mode propagation model. Bayesian op...

By Heng Zhang, Haotian Xiang, Florian Meyer, Qin Lu

benchmarks

arXiv AI

Sep 15

Steering Generative Robot Policies with Lexicographic Preferences

arXiv:2609.15014v1 Announce Type: cross Abstract: Pretrained generative robot policies can produce effective behaviors across diverse environments, but deployment can lead to requirements and prefere...

By Yixuan Jia, Jonathan P. How

diffusion robotics benchmarks

arXiv Machine Learning

Sep 15

GNN4PPM: Multi-Target Predictive Process Monitoring with Relational Graph Convolutional Networks

arXiv:2609.14534v1 Announce Type: new Abstract: Predictive Process Monitoring (PPM) aims at predicting at runtime and as early as possible the future states of a process execution. Common tasks inclu...

By Ana Costa, Johannes M\"akelburg, Luise Pufahl

rag benchmarks

Get the daily briefing