Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 1

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limit...

By Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang

llms agents benchmarks safety

arXiv AI

Oct 1

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable r...

By \.Ibrahim Ethem Deveci, Funda Tan \c{C}al{\i}k, Bar{\i}\c{s} Deniz Sa\u{g}lam, Duygu Ataman

llms reinforcement-learning benchmarks

arXiv AI

Oct 1

AIM: Agentic Idea Management for Automated Research

arXiv:2609.38445v1 Announce Type: new Abstract: Frontier LLMs are increasingly used to automate scientific research through iterative search. We distinguish idea-driven search from solution-driven se...

By Hyeong Kyu Choi, Bhavana Dalvi Mishra, Jiefeng Chen, Mihir Parmar, Rui Meng, Chun-Liang Li, Xiangru Tang, Sharon Li, Jinsung Yoon, Tomas Pfister

llms agents benchmarks safety

arXiv AI

Oct 1

D-GAP: Improving Out-of-Domain Robustness via Dataset-Agnostic and Gradient-Guided Augmentation in Frequency and Pixel Spaces

arXiv:2511.11286v4 Announce Type: replace-cross Abstract: Out-of-domain (OOD) robustness is challenging to achieve in real-world computer vision, especially in unsupervised domain adaptation scenario...

By Ruoqi Wang, Haitao Wang, Shaojie Guo, Qiong Luo

computer-vision fine-tuning benchmarks safety

arXiv AI

Oct 1

Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models

arXiv:2605.07872v2 Announce Type: replace-cross Abstract: Multimodal reward models have advanced substantially in text and image domains, yet progress in video understanding reward modeling remains s...

By Yuancheng Wei, Linli Yao, Lei Li, Haojie Zhang, Hao Zhou, Fandong Meng, Xu Sun

reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes

arXiv:2605.26380v2 Announce Type: replace-cross Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...

By Jingru Chen, Yiming Liu, Mingtao Chen, Sijie Chen, Richeng Xuan, Liang Yang, Zhichao Hu, Fanyang Lu

llms multimodal benchmarks

arXiv Computation and Language

Oct 1

AdaGEPA: Adaptive Feedback Allocation for Reflective Prompt Optimization

arXiv:2609.39927v1 Announce Type: new Abstract: Prompt optimization improves the performance of language-model systems on downstream tasks by refining their prompts. Classical methods evaluate prompt...

By Junyang Chen, Zecheng Wang, Jingbang Chen

benchmarks

arXiv Computation and Language

Oct 1

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

arXiv:2607.05721v2 Announce Type: replace Abstract: Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refin...

By Yimeng Zhang, Yingying Zhuang, Ziyi Wang, Yuxuan Lu, Pei Chen, Aman Gupta, Zhe Su, Ming Tan, Zhilin Zhang, Qun Liu, Manikandarajan Ramanathan, Rajashekar Maragoud, Edward Vul, Jing Huang, Dakuo Wang

llms benchmarks

arXiv AI

Oct 1

Do Self-Evolving Skills Generalize to Held-Out Tasks?

arXiv:2609.39148v1 Announce Type: new Abstract: AI agents can externalize what they learn from past tasks into reusable \emph{skills}, such as procedures, checklists, code, or other executable artifa...

By Xihao Piao, Zifeng Wang, Zhen Chen

llms agents benchmarks

arXiv AI

Oct 1

ANI: Adaptive Numerical Injection for Unifying Semantic and Arithmetic Representations in Numerical Reasoning

arXiv:2609.39294v1 Announce Type: new Abstract: Precise numerical reasoning with Large Language Models (LLMs) is essential for expanding their applicability to complex real-world tasks. However, text...

By Jinsung Jeon, Seung-won Hwang

llms rag diffusion nlp benchmarks

arXiv AI

Oct 1

Divide and Collapse: MAPF-Collapse via Exact Decomposition into Independent Sub-Instances

arXiv:2609.39559v1 Announce Type: new Abstract: In this work we study the problem of MAPFC, a post-optimization step for Multi-Agent Path Finding (MAPF) plans where we are given a feasible plan produ...

By Oren Salzman

agents benchmarks

arXiv AI

Oct 1

SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models

arXiv:2609.38716v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have made remarkable progress across visual perception tasks, yet spatial reasoning remains a persistent weaknes...

By Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu

llms multimodal benchmarks

arXiv AI

Oct 1

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

arXiv:2609.40303v1 Announce Type: new Abstract: Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnati...

By Kirill Brilliantov, Alejandro Hern\'andez-Cano, Emmanuel Abb\'e

llms agents benchmarks

arXiv AI

Oct 1

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

arXiv:2609.38955v1 Announce Type: cross Abstract: Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a...

By Yang chen, Yitan Zhang, Michael Witbrock, Shuyue Hu

diffusion reinforcement-learning benchmarks

arXiv AI

Oct 1

What Streaming Anomaly Detection Finds (and Misses) in Industrial Time Series

arXiv:2609.39232v1 Announce Type: cross Abstract: EDF relies on continuous monitoring of its power plants to detect anomalies as soon as they occur. Given the absence of a universally optimal streami...

By Magali Parrino, Antoine Ajenjo, Emmanuel Remy, Pierre Stephan, Paul Boniol

benchmarks

arXiv AI

Oct 1

MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution

arXiv:2609.38349v1 Announce Type: cross Abstract: Modern agentic systems combine an AI model with a harness that controls execution and environmental interactions. Harness design strongly affects lon...

By Prithwish Jana, Mononito Goswami, Hao Liu, Xinyu Li, Langlin Huang, Zhehui Huang, Zhishen Huang, Patrick Bl\"obaum, Anoop Deoras, Purak Jain, Nikos Kanakaris, Sahika Genc

llms agents benchmarks

arXiv AI

Oct 1

Simulator-Refined Diffusion for Radio-Frequency Inverse Design

arXiv:2609.38363v1 Announce Type: cross Abstract: Diffusion models have shown potential in inverse design of printed circuit boards (PCBs), enabling the generation of layouts conditioned on target S-...

By Jinhao Liang, Jacob K. Christopher, Michael Frei, Tommaso Dreossi, Nando Fioretto

diffusion benchmarks

arXiv Computer Vision

Oct 1

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

arXiv:2609.38463v1 Announce Type: cross Abstract: Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not...

By Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev, Artem Belyaev, Andrey Kuznetsov, Denis Shepelev, Vlad Shakhuro

agents robotics fine-tuning benchmarks

arXiv Computer Vision

Oct 1

TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images

arXiv:2603.07119v3 Announce Type: replace Abstract: Recent text-to-image models have improved global realism, but text rendering remains a persistent failure mode: images may look convincing overall,...

By Kirill Koltsov, Aleksandr Gushchin, Anastasia Antsiferova, Dmitriy Vatolin

diffusion benchmarks safety

arXiv AI

Oct 1

MoFlow: Multi-Objective Agentic Workflow Generation

arXiv:2609.38294v1 Announce Type: new Abstract: We study the generation of agentic workflows that jointly optimize multiple objectives, such as accuracy, cost, latency, robustness, and consistency. E...

By Yining Lu, Aurelie Lozano, Xi Yang, Naoki Abe, Yu Deng, Meng Jiang

agents nlp reinforcement-learning benchmarks

Get the daily briefing