Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

Self-Evolving Visual Agent via Margin Reward

arXiv:2610.15084v1 Announce Type: new Abstract: Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool in...

By Wenxi Yang, Yuzhong Zhao, Baochang Zhang, Yunfan Liu, Qixiang Ye

agents reinforcement-learning multimodal benchmarks

arXiv AI

Oct 15

Using Evidence-Centered Design to Guide Prompt Engineering in Measurement: An Illustration with Automated Essay Scoring

arXiv:2610.15231v1 Announce Type: new Abstract: Generative artificial intelligence (GenAI) models, particularly large language models (LLMs), are increasingly used in educational settings for tasks s...

By Jiawei Xiong, Cheng Tang, Hong Jiao, Hanna Choi

llms benchmarks safety

arXiv AI

Oct 15

ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail

arXiv:2610.15239v1 Announce Type: new Abstract: Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinic...

By Anurag Sharma, Aditya Sharma, Sapna Baalwan

llms multimodal benchmarks

arXiv AI

Oct 15

TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

arXiv:2610.15365v1 Announce Type: new Abstract: Large language models (LLMs) can produce harmful content while generating a response. A useful safeguard should not only detect this risk during genera...

By Ankur Garg, Xuemin Yu, Hassan Sajjad, Samira Ebrahimi Kahou

llms benchmarks

arXiv AI

Oct 15

SkillFusion: Towards Reliable Multi-Skill Composition for LLM Agents

arXiv:2610.15481v1 Announce Type: new Abstract: Independently authored skills can impose incompatible requirements, competing preferences, and redundant instructions when jointly loaded by an LLM age...

By Enhao Huang, Xinye Zhao, Yiming Li, Bingrun Yang, Nanjun Zhou, Chunshu Xia, Hongwei Yao, Zhan Qin

llms agents efficiency benchmarks safety

arXiv AI

Oct 15

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

arXiv:2610.15634v1 Announce Type: new Abstract: Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contrib...

By Marthe Ballon, Arne Vanhoyweghen, Seorin Kim, Francesca Carlon, Lynn Delcon, Melika Mobini, Vincent Holst, Lukas Van de Voorde, Tibo Vanleke, Mathieu Louis, Bert Verbruggen, Brecht F. Verbeken, Andres Algaba, Vincent Ginis

llms benchmarks

arXiv AI

Oct 15

From Stages to States: Unified Iterative Reasoning for MLLMs

arXiv:2610.15620v1 Announce Type: new Abstract: Multimodal Chain-of-Thought (MCoT) substantially enhances the complex reasoning capabilities of Multimodal Large Language Models (MLLMs) by explicitly...

By Chuang Yu, Xuefei Ma, Yuanhao Feng, Wuxiong Zeng, Bo Wang, Minghong Cai, Dongming Wu, Fei Zhou, Chen Tang, Wenwen Tong, Lewei Lu, Xiangyu Yue

llms multimodal benchmarks

arXiv AI

Oct 15

SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications

arXiv:2610.15708v1 Announce Type: new Abstract: Coding agents based on frozen language models can improve their performance by refining the external context they rely on, without updating model weigh...

By Yutong Cheng, Haifeng Chen, Peng Gao, Wei Cheng

llms agents benchmarks

arXiv AI

Oct 15

Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate

arXiv:2610.15710v1 Announce Type: new Abstract: Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair...

By Xupeng Chen, Chenqian Le

llms benchmarks

arXiv AI

Oct 15

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

arXiv:2610.15714v1 Announce Type: new Abstract: Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions un...

By Rui Tang, Hongqiu Wang

agents multimodal benchmarks safety

arXiv AI

Oct 15

Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation

arXiv:2610.15797v1 Announce Type: new Abstract: Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan,...

By Jingjing (May), Liu, Dishu Yang, Hongbo Qin, Hansong Zhang

benchmarks data-engineering

arXiv AI

Oct 15

The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much

arXiv:2610.15938v1 Announce Type: new Abstract: Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown re...

By Nitish Kumar, Prateik Pothuneedi, Greg Bandak

llms nlp benchmarks

arXiv AI

Oct 15

ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation

arXiv:2610.15956v1 Announce Type: new Abstract: Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environ...

By Wenjun Huang, Shenghao Fu, Yian Jin, Yang Ni, Yifan Li, Lichi Li, Xingjian Diao, Pengfei Zhang, Yutong Song, Hanning Chen, Jiarui Cui, Tianyu Yang, Shilong Li, Yu Kong, Mohsen Imani

reinforcement-learning robotics multimodal benchmarks

arXiv AI

Oct 15

GUIDE: Computationally Augmented Consensus for Trustworthy Clinical Practice Guidelines

arXiv:2610.16108v1 Announce Type: new Abstract: Maintaining up-to-date clinical practice guidelines (CPGs) is essential for evidence-based medicine, but rapid evidence growth and complex expert conse...

By Yuanfan Lin, Jie Di, Xujin He, Zhenzhe Lin, Jialing Chen, Ruiyang Li, Changyu Zheng, Yuanrui Shang, Weiqin Peng, Jingxi Qin, Xiong Wang, Dongxin Ye, Yining Huang, Guoqiang Li, Andi Xu, Sijia Yu, Ruixin Wang, Yahan Yang, Xiaohang Wu, Duoru Lin, Jingjing Chen, Zhenzhen Liu, Wenben Chen, Haotian Lin

agents benchmarks safety

arXiv AI

Oct 15

HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

arXiv:2610.16319v1 Announce Type: new Abstract: Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluat...

By Zijian Cai, Haowei Liu, Kailai Chen, Lin Peng, Cong Wan, Ziqi Zhang, Chunfeng Yuan, Bing Li, Weiming Hu, Haiyang Xu

agents benchmarks

arXiv AI

Oct 15

Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents

arXiv:2610.16998v1 Announce Type: new Abstract: Deep search enables language agents to solve complex information-seeking tasks through iterative web exploration. However, developing robust open-sourc...

By Cong Pang, Jiaqi Su, Chengrui Wu, Kuihao Yan, Tianrui Zhang, Zixuan Qiu, Chao Lin, Zixiu Lin, Zixuan Chen, Zelin Xiao, Tiankuo Yao, Jiawei Hong, Zixuan Chen, Dan Xu, Lewei Lu, Wenjie Wang, Xin Lou

agents nlp benchmarks

arXiv AI

Oct 15

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

arXiv:2610.17183v1 Announce Type: new Abstract: Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an o...

By Aotian Chen, Frazier N. Baker, Reza Averly, Shan Zhang, Tianao Chen, Daniel Adu-Ampratwum, Xia Ning, Yuan Xue

llms agents multimodal benchmarks

arXiv AI

Oct 15

Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce

arXiv:2610.17378v1 Announce Type: new Abstract: AI agents now browse and buy on people's behalf, and benchmarks measure how well the agents perform. Far less is known about the other side of the tran...

By Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim, Ibrahim Shaer, Tamaz Andguladze

agents benchmarks

arXiv AI

Oct 15

OpenDepth: Repurposing Image Editing Priors for Robust Open-World Monocular Depth Estimation

arXiv:2610.17610v2 Announce Type: new Abstract: Recently, the integration of generative priors has revolutionized Monocular Depth Estimation (MDE), achieving state-of-the-art (SOTA) performance with...

By Mengke Zhang, Tianyi Yang, Xinzhuo Li, Anlong Ming

llms diffusion computer-vision benchmarks safety

arXiv AI

Oct 15

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

arXiv:2610.17609v1 Announce Type: new Abstract: Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how...

By Tianyi Xu, Yaolun Zhang, Yiran Wu, Joe Nguyen, Shengyu Dai, Saba Moeinizade, Zhenwen Shao, Qingyun Wu, Huazheng Wang

agents benchmarks

Get the daily briefing