Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial...

By Noam Slonim, Aviv Slobodkin, Gil Fidel, Gal Moshitch, Mimi Sun, Lily Mihalkova, Joydeep Paul, Per Bjornsson, Sydney Lin, Jamie McPike, Samapriya Roy, Utkarsh Patel, Gia Jung, Zachary Luna, Allison Berman, William Cai, Xiao Wu, Mohit Agarwal, Mandar Sharma, Aliza Hoffman, Mark Young, Rama Pasumarthi, Roy Lee, Ruth Alcantara, Luke Barrington, Aaron Bell, Reuven Sayag, Avinatan Hassidim, Yossi Matias, Tyler McHenry, Ronald Ho, Christopher Haire, Oliver Guinan, Niv Efron, Tim Thelin, Gautam Prasad, Shravya Shetty

llms agents benchmarks

arXiv AI

Oct 15

Comparative Study of Canvas- and Coordinate-Based Geometry-Conditioned Neural Networks for Real-Time Prediction of Indoor Environments with Varying Room Dimensions

arXiv:2610.15885v1 Announce Type: new Abstract: Computational fluid dynamics (CFD) is computationally expensive for iterative building design. Deep learning surrogates offer faster predictions, but g...

By Shaofan Wang, Philipp Geyer

benchmarks data-engineering

arXiv Machine Learning

Oct 15

X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking

arXiv:2606.22278v2 Announce Type: replace-cross Abstract: Ensuring safety of learning-enabled robotic manipulation across diverse embodiments and tasks still requires significant manual engineering....

By Alex Beaudin, Hanna Krasowski, Kartik Nagpal, Sanjit A. Seshia, Murat Arcak, Negar Mehr

robotics benchmarks safety

arXiv Computation and Language

Oct 15

Learning Reusable Agent Subroutines from Experience

arXiv:2606.02994v2 Announce Type: replace-cross Abstract: Existing methods induce various forms of knowledge from agent traces, from sample workflows to programmatic skills. We propose ARISE (Agent R...

By Zhihan Lei, Jiarui Yan, Joshua Momo, William W. Cohen

llms agents benchmarks

arXiv AI

Oct 15

Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing

arXiv:2610.15993v1 Announce Type: new Abstract: We study how initial model selection, later handoff, and training targets affect success and cost in LLM tasks that span multiple turns. We compare sev...

By Daniel Matos, Lu Liu, Yanfeng Zhang

llms benchmarks

arXiv Computation and Language

Oct 15

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

arXiv:2610.19934v1 Announce Type: new Abstract: Weight pruning can degrade the accuracy of large reasoning models (LRMs), yet it remains unclear whether it damages reasoning itself. We find that much...

By Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu, Dung D. Le, Wei Lu

efficiency benchmarks

arXiv AI

Oct 15

LLMRec: Benchmarking Large Language Models on Recommendation Task

arXiv:2308.12241v2 Announce Type: replace-cross Abstract: Recently, the fast development of Large Language Models (LLMs) such as ChatGPT has significantly advanced NLP tasks by enhancing the capabili...

By Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, Philip S. Yu

llms nlp fine-tuning benchmarks

arXiv AI

Oct 15

ClawLoop: Self-Improving Agents via Harness-Native Training Loop

arXiv:2610.14118v1 Announce Type: new Abstract: The rapid advancement of Large Language Models (LLMs) has driven their deployment toward general-purpose harness environments, where LLM-based agents o...

By Hansi Yang, Linji Hao, Hongyan Hao, An Wang, Han Wu, Qi Gu, Xunliang Cai

llms agents reinforcement-learning fine-tuning benchmarks data-engineering

arXiv AI

Oct 15

VETNET: Assembling Benchmark Datasets for AI-Driven Veterinary Innovation

arXiv:2610.14149v1 Announce Type: new Abstract: VETNET represents an ecosystem of live benchmark datasets and evaluation frameworks designed to accelerate AI-driven innovation in animal science and v...

By Katherine J. Koebel, Renata Ivanek, Parminder Basran, Jennifer Sun, Casey Cazer, Gayatri Anil, Tommaso Banzato, Liz Barton, Dottie Cimino Brown, Leszek Bukowski, Marta Castelhano, Simon Coghlan, Christopher De Sa, Nikki Dutta, Hala Ghali, Julio Giordano, Joshua Glass, David C. Hogg, Nicholas Hollingshead, Adam Kiehl, Ziang Liu, Monica Meduri, Param Mohapatra, Kim Nayyer, Amelia Nelson, Peter-John Mantyla Noble, Christopher Pinard, Beryl Rabindran, Scott Ross, Ron Seccia, Moira Sheehan, Fei Wang, Tracy L. Webb, Kilian Q. Weinberger, Heather M. Whitney, Chen Xin

benchmarks

arXiv AI

Oct 15

Unconstrained CTI-REALM with Open-Weights

arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and...

By Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma

llms agents benchmarks

arXiv AI

Oct 15

From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

arXiv:2506.23101v2 Announce Type: replace-cross Abstract: Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in sociall...

By Yue Xu, Wenjie Wang

llms multimodal benchmarks safety

arXiv AI

Oct 15

When and How to Canonize: A Generalization Perspective

arXiv:2605.11008v3 Announce Type: replace-cross Abstract: While invariant architectures are standard for processing symmetric data, there is growing interest in achieving invariance by applying group...

By Yonatan Sverdlov, Benjamin Friedman, Snir Hordan, Nadav Dym

benchmarks

arXiv AI

Oct 15

A Hybrid Kinematic GCN and Elastic Transformer for Efficient Indian Sign Language Recognition

arXiv:2610.14680v1 Announce Type: new Abstract: Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language...

By Pushp Garg, Himangshu Sarma, Mrinmoy Ghorai

llms benchmarks

arXiv AI

Oct 15

PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation

arXiv:2512.23994v5 Announce Type: replace-cross Abstract: Text-to-audio-video (T2AV) generation is central to applications such as filmmaking and world modeling. However, current models often fail to...

By Tianxin Xie, Wentao Lei, Kai Jiang, Guanjie Huang, Pengfei Zhang, Chunhui Zhang, Fengji Ma, Haoyu He, Han Zhang, Jiangshan He, Jinting Wang, Linghan Fang, Lufei Gao, Orkesh Ablet, Peihua Zhang, Ruolin Hu, Shengyu Li, Weilin Lin, Xiaoyang Feng, Xinyue Yang, Yan Rong, Yanyun Wang, Zihang Shao, Zelin Zhao, Chenxing Li, Shan Yang, Wenfu Wang, Meng Yu, Dong Yu, Li Liu

diffusion multimodal benchmarks

arXiv AI

Oct 15

CESNET-TimeSeries26: A Multi-Year Network Traffic Time Series Dataset for Annual Seasonality, Concept Drift, and Long-Term Anomaly Detection

arXiv:2610.14799v1 Announce Type: new Abstract: Forecast-based models are fundamental to network traffic management and anomaly detection. However, it is essential to evaluate these forecasting model...

By Josef Koumar, Karel Hynek, Kamil Jerabek, Pavel Siska

benchmarks

arXiv AI

Oct 15

Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems

arXiv:2605.29790v2 Announce Type: replace-cross Abstract: LLM-based multi-agent systems (MAS) have emerged as an effective paradigm for complex and long-horizon tasks. However, in real-world tasks, M...

By Zhezheng Hao, Tianfu Wang, Huanshuo Dong, Ziyan Liu, Hong Wang, Xiankun Lin, Qiang Lin, Can Wang, Hande Dong, Jiawei Chen

llms agents benchmarks

arXiv AI

Oct 15

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

arXiv:2610.15129v1 Announce Type: new Abstract: Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to sp...

By Zonghuan Li, Hayley Hung, Chirag Raman

llms multimodal benchmarks

arXiv AI

Oct 15

Harmony Harness: Multi-Objective Self-Evolution of Agent Harnesses

arXiv:2610.15150v1 Announce Type: new Abstract: Recent work on self-evolving LLM agents treats the agent harness as an optimization target that can be iteratively improved through execution feedback....

By Weichen Yu, Lesly Miculicich, Bhavana Dalvi Mishra, Jiefeng Chen, Shuo Li, Matt Fredrikson, Tomas Pfister, Long T. Le

llms agents benchmarks

arXiv AI

Oct 15

Process Advantage Signal Shaping: Advantage Construction for Process-Supervised Reinforcement Learning in LLM Reasoners

arXiv:2606.29296v2 Announce Type: replace Abstract: Dense process supervision supplies local feedback but leaves open how that feedback should become a policy advantage. We introduce PASS (Process Ad...

By Chao Wang, Hongtao Tian, Tao Yang, Yunsheng Shi, Ting Yao, Wenbo Ding

llms nlp reinforcement-learning efficiency benchmarks

arXiv AI

Oct 15

Eval4DiRec: A Unified and Systematic Evaluation Framework for Diffusion-based Recommender Systems

arXiv:2609.34404v2 Announce Type: replace-cross Abstract: Leveraging the strong generative capabilities and stable training dynamics of diffusion models, diffusion-based recommender systems (RSs) hav...

By Cong Wang, Shoujin Wang, Yishuo Li, Qi Zhang, Liang Hu, Wenpeng Lu

diffusion benchmarks

Get the daily briefing