Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 23

From Code to Requirements: Agentic Reverse Engineering of Business Rules at Enterprise Scale

arXiv:2609.22719v1 Announce Type: cross Abstract: Business requirements for enterprise software systems are rarely captured in structured form; the logic resides instead in source code, configuration...

By Garima Agrawal, Prasun Das, Priyanka L, Minisha N, Akshay Sarvade, Hemath Manivanan, Sravani Joshna, Prasad Kalyansundaram

agents benchmarks

arXiv AI

Sep 23

Commonsense-Grounded Path Planning from Abstract Instructions

arXiv:2609.22813v1 Announce Type: cross Abstract: We present \emph{commonsense ranked search} (CoRS), a novel path planner that turns an abstract instruction into a route that follows commonsense. Wh...

By Masafumi Endo, Kohei Honda, Ryo Yonetani

llms robotics multimodal benchmarks

arXiv AI

Sep 23

Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs

arXiv:2609.22851v2 Announce Type: cross Abstract: Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio...

By Jing Peng, Zichao Nie, Zhisheng Zhang, Jingran Xie, Zhiyong Wu

llms nlp benchmarks

arXiv AI

Sep 23

NostrAgent: A Decentralized Identity and Delegation Architecture for Sovereign Agentic Systems

arXiv:2609.22944v1 Announce Type: cross Abstract: Autonomous AI agents increasingly act across organizational boundaries on behalf of human operators: they invoke third-party services, delegate subta...

By Oliver Aleksander Larsen, Mahyar Tourchi Moghaddam

agents benchmarks

arXiv AI

Sep 23

Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models

arXiv:2609.23589v1 Announce Type: cross Abstract: Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio repr...

By Jiaheng Dong, Xiaofeng Yu, Jean Honorio, Abhirup Ghosh, Hong Jia, Ting Dang

llms reinforcement-learning multimodal benchmarks

arXiv AI

Sep 23

Controlling risks of AI in chemical science with agents

arXiv:2312.06632v2 Announce Type: replace Abstract: Artificial intelligence is rapidly advancing scientific discovery, but this progress carries risks of misuse, such as the creation of harmful subst...

By Jiyan He, Haoxiang Guan, Weitao Feng, Yaosen Min, Jingwei Yi, Kunsheng Tang, Shuai Li, Jie Zhang, Kejiang Chen, Wenbo Zhou, Xing Xie, Weiming Zhang, Nenghai Yu, Shuxin Zheng

llms agents benchmarks safety

arXiv AI

Sep 23

Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models

arXiv:2510.21175v2 Announce Type: replace Abstract: Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range...

By Yujin Jo, Taesup Kim

llms efficiency multimodal benchmarks

arXiv AI

Sep 23

VeRA: Renewing Reasoning Benchmarks with Executable Specifications

arXiv:2602.13217v2 Announce Type: replace Abstract: Reasoning benchmarks need renewal along two axes: freshness and headroom. VeRA makes both executable and auditable by turning each item into a task...

By Zerui Cheng, Jiashuo Liu, Chunjie Wu, Jiayang Sun, Jianzhu Yao, Pramod Viswanath, Ge Zhang, Wenhao Huang

benchmarks

arXiv AI

Sep 23

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

arXiv:2605.01457v4 Announce Type: replace Abstract: How can generative offline multi-agent reinforcement learning achieve both fast joint trajectory generation and effective cooperation? Multi-agent...

By Guowei Zou, Haitao Wang, Beiwen Zhang, Boning Zhang, Hejun Wu

agents diffusion reinforcement-learning efficiency benchmarks

arXiv AI

Sep 23

LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories

arXiv:2608.15242v4 Announce Type: replace Abstract: In long agent executions, an early error can persist through later actions and checks, while evidence needed to trace its origin is dispersed acros...

By Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma, Jiayi Zhang, Yongzhou Yao, Yanan Zhao, Fei Sun, Yintong Huo, Zhaoyang Liu, Jingjing Li, Gaogang Xie, Dan Pei

agents benchmarks

arXiv AI

Sep 23

ADSEL: Adaptive Dual Self-Expression Learning for EEG Feature Selection via Incomplete Multi-Dimensional Emotion Labels

arXiv:2508.05229v2 Announce Type: replace-cross Abstract: EEG based multi-dimension emotion recognition has attracted substantial research interest in affective computing. However, the high dimension...

By Xueyuan Xu, Tianze Yu, Junming Zhang, Chenrui Liu, Wenjia Dong, Fulin Wei, Li Zhuo

benchmarks

arXiv AI

Sep 23

Learning Energy-Efficient Air--Ground Actuation for Hybrid Robots on Stair-Like Terrain

arXiv:2603.26687v2 Announce Type: replace-cross Abstract: Hybrid aerial--ground robots can use thrust to cross obstacles that impede wheel-driven motion, but deciding how much thrust to apply during...

By Jiaxing Li, Ishaan Bhimwal, Wen Tian, Xinhang Xu, Junbin Yuan, Yuxin Guo, Sebastian Scherer, Muqing Cao

reinforcement-learning robotics benchmarks

arXiv AI

Sep 23

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

arXiv:2609.12708v2 Announce Type: replace-cross Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whe...

By Cristina Improta, Pietro Liguori, Domenico Cotroneo

llms benchmarks safety

arXiv AI

Sep 23

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

arXiv:2609.15387v3 Announce Type: replace-cross Abstract: Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automat...

By Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao, Zhexin Zhang, Guang Chen, Haowei Lin, Ying Zhou, Tianyi Bai, Dolly Deng, Suncong Zheng, Maxm Pan

llms agents benchmarks safety

arXiv Machine Learning

Sep 23

Federating Quantum and Classical Computing: A Privacy-Preserving Hybrid Approach

arXiv:2609.25082v1 Announce Type: new Abstract: Quantum machine learning (QML) is increasingly recognized as one of the most promising near-term applications of quantum computing, viewed as a next-fr...

By Carlos Cano, Daniel M. Jimenez-Gutierrez, Diego Sal, Georgios Kellaris, Joaquin del Rio, Oleksii Sliusarenko, Xabi Uribe-Etxebarria

rag computer-vision benchmarks safety

arXiv Machine Learning

Sep 23

Multi-Term Fourier Graph Neural Network with Sample Relationship Learning for Enhanced Remaining Useful Life Prediction

arXiv:2609.25179v1 Announce Type: new Abstract: Predicting the remaining useful life (RUL) is essential for effective predictive maintenance. Spatio-Temporal Graph Neural Networks (ST-GNNs), which ca...

By Ya Song, Laurens Bliek, Yaoxin Wu, Yingqian Zhang

benchmarks

arXiv Machine Learning

Sep 23

Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining

arXiv:2609.25482v1 Announce Type: new Abstract: Large language model (LLM) pretraining conventionally returns the raw final iterate. This couples two design choices: the learning-rate schedule that g...

By Adam Ousherovitch, Yixin Wang

llms benchmarks

arXiv Machine Learning

Sep 23

SambaGraph: Action-Reaction Spatio-Temporal Graphs for Soccer Tactical Response Modeling

arXiv:2609.25569v1 Announce Type: new Abstract: Soccer tactics are interactive: an attacking action changes the opponent's defensive problem, and the observed response depends on the multi-agent matc...

By Abel A. Reyes-Angulo, Henry O. Velesaca, Steven Araujo

llms rag agents benchmarks

arXiv Machine Learning

Sep 23

Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs

arXiv:2609.25602v1 Announce Type: new Abstract: In language models, the choice between believing the prompt and believing the weights is made by a handful of identifiable attention heads. Instruction...

By Shubham Santosh Pandere, Gautam Ranka, Ritika Varshney, Navya Deshmukh, Roushni Sareen, Roshan Kumar Singh

llms fine-tuning benchmarks safety

arXiv Machine Learning

Sep 23

Minimal Recurrent Behavioral Memory for Imitation under Partial Observability

arXiv:2609.25757v1 Announce Type: new Abstract: What is the least recurrent memory needed to reproduce a specified expert under partial observability? The instantaneous requirement is the conditional...

By Xianyao Li, Fang Xu, Rui Min, Ruitong Tian, Jing Du

robotics benchmarks

Get the daily briefing