Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 15

Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

arXiv:2607. 11969v1 Announce Type: cross Abstract: Point-adjustment (PA), long the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al.

By Zongye Lyu

benchmarks safety

arXiv AI

Jul 15

1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World

arXiv:2602. 18548v2 Announce Type: replace-cross Abstract: Design-to-code translates high-fidelity UI designs into executable front-end implementations, but progress remains hard to compare due to inconsistent datasets, toolchains, and evaluation protocols.

By Qiao Xu, Yipeng Yu, Chengxiao Feng, Xu Liu

reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jul 15

Repairing Shape-Prior Shortcuts in Long-Range Single-Shot Fringe Projection Profilometry

arXiv:2607. 11928v1 Announce Type: new Abstract: Single-shot fringe projection profilometry (FPP) networks that regress depth directly can exploit a shape-prior shortcut, recovering depth from object boundaries rather than from fringe phase.

By Adam Haroon, Cody Fleming, Beiwen Li

benchmarks safety

arXiv AI

Jul 15

Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning

arXiv:2512. 01113v2 Announce Type: replace-cross Abstract: Algorithmic reasoning -- the ability to perform step-by-step logical inference -- is a synthetic benchmark for evaluating multi-step reasoning abilities, designed for graph neural networks and also for transformer models.

By Dongyue Li, Zhenshuo Zhang, Minxuan Duan, Edgar Dobriban, Hongyang R. Zhang

llms benchmarks

arXiv AI

Jul 15

Good Benchmarks

arXiv:2607. 12217v1 Announce Type: new Abstract: Good tasks are correct, solvable, verifiable, well-specified, and hard for interesting reasons.

By Ivan Bercovich

benchmarks

arXiv Machine Learning

Jul 15

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

arXiv:2607. 12278v1 Announce Type: cross Abstract: Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answering (VQA) benchmarks.

By Wenhao Zhang, Zhongliang Zhou, John Kang, Sheng Li

llms nlp multimodal benchmarks

arXiv AI

Jul 15

Generalized Distribution-Free Semi-Supervised Learning with Risk Rewrite

arXiv:2607. 11947v1 Announce Type: cross Abstract: Typical semi-supervised learning (SSL) methods rely on distributional assumptions, and their performance degrades when these are violated.

By Yushi Hirose, Hiroo Irobe, Takafumi Kanamori

benchmarks

arXiv AI

Jul 15

Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

arXiv:2607. 12273v1 Announce Type: cross Abstract: As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences.

By Xiaoning Ren, Yinxing Xue, Lei Ma, Yuheng Huang

llms rag benchmarks safety

arXiv Machine Learning

Jul 15

AnySleep: a channel-agnostic deep learning system for high-resolution sleep staging in multi-center cohorts

arXiv:2512. 14461v2 Announce Type: replace Abstract: Sleep is essential for health, yet studying its dynamics requires manual sleep staging, a labor-intensive step in research and clinical care.

By Niklas Grieger, Jannik Raskob, Siamak Mehrkanoon, Stephan Bialonski

benchmarks

arXiv AI

Jul 15

Toward Trustworthy Autonomous Science: A Two-Year Community Roadmap

arXiv:2607. 12113v1 Announce Type: cross Abstract: One year ago, the AISLE roadmap argued that autonomous laboratories operated as isolated islands and proposed a grassroots network organized around five critical dimensions.

By Rafael Ferreira da Silva, Milad Abolhasani, Peter Beaucage, Laura Biven, Michael Bussmann, Kyle Chard, Ryan Coffee, Stephen DeWitt, Sagar Dolas, Carrie Eckert, David Elbert, Ian Foster, Tirthankar Ghosal, Anna Giannakou, Tom Gibbs, Leslie Hamilton, Glenn Lockwood, Theresa Mayer, Ben Mintz, Raffi Nazikian, Sal Nimer, Amanda Randles, Woong Shin, Sreenivas Rangan Sukumar, Fr\'ed\'eric Suter, Mitra Taheri, Michela Taufer, Draguna Vrabie

agents robotics benchmarks safety

arXiv AI

Jul 15

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

arXiv:2607. 12787v1 Announce Type: new Abstract: Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc.

By Kaiwen Zheng, Junchen Fu, Wenhao Deng, Hu Han, Joemon M. Jose, Xuri Ge

llms robotics efficiency multimodal benchmarks safety

arXiv Machine Learning

Jul 15

Verifier-Based Reinforcement Fine-Tuning of Reasoning Models for Thermal Energy Storage Control

arXiv:2607. 12856v1 Announce Type: new Abstract: Buildings are expected to shift cooling loads in response to grid conditions.

By Takumi Shioda, Kohei Terashima, Tatsuo Nagai

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 15

Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence

arXiv:2607. 12916v1 Announce Type: new Abstract: In this work, we introduce CoCo, a loss function aimed at learning normalized and well-structured representations.

By Blanca Cano-Camarero, \'Angela Fern\'andez-Pascual, Jos\'e R. Dorronsoro

rag benchmarks

arXiv AI

Jul 15

Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability

arXiv:2602. 11198v2 Announce Type: replace-cross Abstract: Multi-agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code.

By Shafiuddin Rehan Ahmed, Sourabh Deshpande

llms agents benchmarks safety

arXiv AI

Jul 15

Text-Aided Multi-Modal Panoptic Symbol Spotting for CAD Floor Plan Drawings

arXiv:2607. 12678v1 Announce Type: cross Abstract: Computer-Aided Design (CAD) floor plan drawings contain both graphical primitives and textual annotations, which provide complementary geometric and semantic cues for intelligent design understanding.

By Yan Gong, Bohao Li, Bowen Du, Junchen Ye

multimodal benchmarks safety

arXiv AI

Jul 15

ReflectWorld-MM: An Entity-Oriented Multimodal Memory System for Open-Ended Video Streams

arXiv:2607. 09759v2 Announce Type: replace-cross Abstract: Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-standing goal, and recently multimodal agents equipped with long-term memory over video streams have attracted increasing interest.

By Xiaokang Ma, Yifan Sun, Zhihong Jin, Jie Gu, Yudong Luo, Shenyi Shao, Chu Tang, Jingmin Chen, Li Pu

agents multimodal benchmarks data-engineering

arXiv Machine Learning

Jul 15

The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography

arXiv:2312. 17670v5 Announce Type: replace-cross Abstract: The Circle of Willis (CoW) is an important network of arteries connecting major circulations of the brain.

By Kaiyuan Yang, Fabio Musio, Yihui Ma, Norman Juchler, Johannes C. Paetzold, Rami Al-Maskari, Luciano H\"oher, Hongwei Bran Li, Ibrahim Ethem Hamamci, Anjany Sekuboyina, Suprosanna Shit, Houjing Huang, Chinmay Prabhakar, Ezequiel de la Rosa, Bastian Wittmann, Diana Waldmannstetter, Florian Kofler, Fernando Navarro, Martin J. Menten, Ivan Ezhov, Daniel Rueckert, Iris N. Vos, Ynte M. Ruigrok, Birgitta K. Velthuis, Hugo J. Kuijf, Pengcheng Shi, Wei Liu, Ting Ma, Maximilian R. Rokuss, Yannick Kirchhoff, Fabian Isensee, Klaus Maier-Hein, Chengcheng Zhu, Huilin Zhao, Philippe Bijlenga, Julien H\"ammerli, Catherine Wurster, Laura Westphal, Jeroen Bisschop, Elisa Colombo, Hakim Baazaoui, Hannah-Lea Handelsmann, Andrew Makmur, James Hallinan, Amrish Soundararajan, Benedikt Wiestler, Jan S. Kirschke, Evamaria O. Riedel, Roland Wiest, Emmanuel Montagnon, Laurent Letourneau-Guillon, Kwanseok Oh, Dahye Lee, Orhun Utku Aydin, Adam Hilbert, Jana Rieger, Dimitrios Rallios, Satoru Tanioka, Alexander Koch, Dietmar Frey, Abdul Qayyum, Moona Mazher, Steven Niederer, Nico Disch, Julius C. Holzschuh, Dominic LaBella, Francesco Galati, Daniele Falcetta, Maria A. Zuluaga, Chaolong Lin, Haoran Zhao, Zehan Zhang, Minghui Zhang, Xin You, Hanxiao Zhang, Guang-Zhong Yang, Yun Gu, Sinyoung Ra, Jongyun Hwang, Hyunjin Park, Junqiang Chen, Marek Wodzinski, Henning M\"uller, Nesrin Mansouri, Florent Autrusseau, Cansu Yalcin, Rachika E. Hamadache, Clara Lisazo, Joaquim Salvi, Adri\`a Casamitjana, Xavier Llad\'o, Uma Maria Lal-Trehan Estrada, Valeriia Abramova, Luca Giancardo, Arnau Oliver, Paula Casademunt, Adrian Galdran, Matteo Delucchi, Oscar Camara, Jialu Liu, Haibin Huang, Yue Cui, Zehang Lin, Yusheng Liu, Shunzhi Zhu, Tatsat R. Patel, Adnan H. Siddiqui, Vincent M. Tutino, Maysam Orouskhani, Huayu Wang, Mahmud Mossa-Basha, Yuki Sato, Sven Hirsch, Susanne Wegener, Bjoern Menze

computer-vision benchmarks

arXiv AI

Jul 15

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

arXiv:2607. 12790v1 Announce Type: new Abstract: Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists.

By Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He

llms agents benchmarks safety data-engineering

arXiv Machine Learning

Jul 15

APPLV: Adaptive Planner Parameter Learning from Vision-Language-Action Model

arXiv:2603. 08862v2 Announce Type: replace-cross Abstract: Autonomous navigation in highly constrained environments remains challenging for mobile robots.

By Yuanjie Lu, Beichen Wang, Zhengqi Wu, Yang Li, Xiaomin Lin, Chengzhi Mao, Xuesu Xiao

llms agents reinforcement-learning robotics fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 15

Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs

arXiv:2607. 12985v1 Announce Type: new Abstract: Aligned language models routinely misreport under non-evidential incentive pressure: they agree with a confident user or overstate certainty even when their internal belief is unchanged.

By Sen Yang, Yuen-Hei Yeung

llms fine-tuning benchmarks

Get the daily briefing