Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 15

From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness

arXiv:2607. 12166v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are the standard for decomposing superposed neural representations into interpretable features, and evaluation relies predominantly on correlational recovery metrics -- cosine similarity between ground-truth directions and decoder atoms.

By Mohamed Abdessalem Bal

efficiency benchmarks safety

arXiv AI

Jul 15

From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

arXiv:2605. 10834v2 Announce Type: replace Abstract: AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets.

By Pedro Conde, Henrique Branquinho, Valerio Mazzone, Bruno Mendes, Andr\'e Baptista, Nuno Moniz

llms agents benchmarks

arXiv Machine Learning

Jul 15

The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography

arXiv:2312. 17670v5 Announce Type: replace-cross Abstract: The Circle of Willis (CoW) is an important network of arteries connecting major circulations of the brain.

By Kaiyuan Yang, Fabio Musio, Yihui Ma, Norman Juchler, Johannes C. Paetzold, Rami Al-Maskari, Luciano H\"oher, Hongwei Bran Li, Ibrahim Ethem Hamamci, Anjany Sekuboyina, Suprosanna Shit, Houjing Huang, Chinmay Prabhakar, Ezequiel de la Rosa, Bastian Wittmann, Diana Waldmannstetter, Florian Kofler, Fernando Navarro, Martin J. Menten, Ivan Ezhov, Daniel Rueckert, Iris N. Vos, Ynte M. Ruigrok, Birgitta K. Velthuis, Hugo J. Kuijf, Pengcheng Shi, Wei Liu, Ting Ma, Maximilian R. Rokuss, Yannick Kirchhoff, Fabian Isensee, Klaus Maier-Hein, Chengcheng Zhu, Huilin Zhao, Philippe Bijlenga, Julien H\"ammerli, Catherine Wurster, Laura Westphal, Jeroen Bisschop, Elisa Colombo, Hakim Baazaoui, Hannah-Lea Handelsmann, Andrew Makmur, James Hallinan, Amrish Soundararajan, Benedikt Wiestler, Jan S. Kirschke, Evamaria O. Riedel, Roland Wiest, Emmanuel Montagnon, Laurent Letourneau-Guillon, Kwanseok Oh, Dahye Lee, Orhun Utku Aydin, Adam Hilbert, Jana Rieger, Dimitrios Rallios, Satoru Tanioka, Alexander Koch, Dietmar Frey, Abdul Qayyum, Moona Mazher, Steven Niederer, Nico Disch, Julius C. Holzschuh, Dominic LaBella, Francesco Galati, Daniele Falcetta, Maria A. Zuluaga, Chaolong Lin, Haoran Zhao, Zehan Zhang, Minghui Zhang, Xin You, Hanxiao Zhang, Guang-Zhong Yang, Yun Gu, Sinyoung Ra, Jongyun Hwang, Hyunjin Park, Junqiang Chen, Marek Wodzinski, Henning M\"uller, Nesrin Mansouri, Florent Autrusseau, Cansu Yalcin, Rachika E. Hamadache, Clara Lisazo, Joaquim Salvi, Adri\`a Casamitjana, Xavier Llad\'o, Uma Maria Lal-Trehan Estrada, Valeriia Abramova, Luca Giancardo, Arnau Oliver, Paula Casademunt, Adrian Galdran, Matteo Delucchi, Oscar Camara, Jialu Liu, Haibin Huang, Yue Cui, Zehang Lin, Yusheng Liu, Shunzhi Zhu, Tatsat R. Patel, Adnan H. Siddiqui, Vincent M. Tutino, Maysam Orouskhani, Huayu Wang, Mahmud Mossa-Basha, Yuki Sato, Sven Hirsch, Susanne Wegener, Bjoern Menze

computer-vision benchmarks

arXiv AI

Jul 15

An Empirical Study for Android-to-OpenHarmony GUI Test Migration

arXiv:2607. 11245v2 Announce Type: replace-cross Abstract: To reduce the substantial engineering effort required to test the corresponding applications from Android to OpenHarmony, migrating existing GUI test cases has become a critical problem.

By Yakun Zhang, Xinjia Chen, Yiyun Chen, Yuxia Zhang, Mingyi Zhou, Xiang Gao, Shaokun Zhang, Li Li, Yunming Ye

benchmarks

arXiv AI

Jul 15

Text-Aided Multi-Modal Panoptic Symbol Spotting for CAD Floor Plan Drawings

arXiv:2607. 12678v1 Announce Type: cross Abstract: Computer-Aided Design (CAD) floor plan drawings contain both graphical primitives and textual annotations, which provide complementary geometric and semantic cues for intelligent design understanding.

By Yan Gong, Bohao Li, Bowen Du, Junchen Ye

multimodal benchmarks safety

arXiv AI

Jul 15

Evidence-Grounded AI for Musculoskeletal Care

arXiv:2607. 12527v1 Announce Type: new Abstract: Musculoskeletal diseases are among the leading causes of disability worldwide and create the greatest global need for rehabilitation.

By Wenjie Li, Yujie Zhang, Fanrui Zhang, Haoran Sun, Renhao Yang, Junjun He, Weiran Huang, Yuanfeng Ji, Chenrun Wang, Kailing Wang, Hongcheng Gao, Kaipeng Zhang, Hanyu Wang, Angela Lin Wang, Xingqi He, Yilin Huang, Shiyi Yao, Lilong Wang, Yankai Jiang, Yirong Chen, Chenglong Ma, Jiyao Liu, Ming Hu, Gen Li, Yidong Xu, Chengyu Zhuang, Jiawei Liu, Yin Zhang, Lequan Yu, Lu Chen, Yinpeng Dong, Lei Liu, Carlos Gutierrez Sanroman, Yu Qiao, Weijie Ma, Xiaosong Wang, Lei Wang

llms benchmarks

arXiv Machine Learning

Jul 15

APPLV: Adaptive Planner Parameter Learning from Vision-Language-Action Model

arXiv:2603. 08862v2 Announce Type: replace-cross Abstract: Autonomous navigation in highly constrained environments remains challenging for mobile robots.

By Yuanjie Lu, Beichen Wang, Zhengqi Wu, Yang Li, Xiaomin Lin, Chengzhi Mao, Xuesu Xiao

llms agents reinforcement-learning robotics fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jul 15

When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?

arXiv:2510. 20963v2 Announce Type: replace Abstract: Multi-agent debate (MAD) was proposed as a promising approach for ensembling the wisdom of multiple large language models (LLMs) to improve reasoning and provide effective supervision to superhuman LLMs.

By Yongqiang Chen, Gang Niu, James Cheng, Bo Han, Masashi Sugiyama

llms agents benchmarks

arXiv Machine Learning

Jul 15

Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctors

arXiv:2512. 15748v2 Announce Type: replace Abstract: Visual Species Recognition (VSR) is a fundamental task in scientific disciplines that require species-level identification, including ecology, palynology, evolutionary biology, systematics, and phylogenetics.

By Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

llms multimodal benchmarks

arXiv AI

Jul 15

The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning

arXiv:2607. 12177v1 Announce Type: new Abstract: The analysis of satellite and aerial imagery has entered a new era with the advent of foundation models.

By Shelley Cazares

llms agents fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 15

AnySleep: a channel-agnostic deep learning system for high-resolution sleep staging in multi-center cohorts

arXiv:2512. 14461v2 Announce Type: replace Abstract: Sleep is essential for health, yet studying its dynamics requires manual sleep staging, a labor-intensive step in research and clinical care.

By Niklas Grieger, Jannik Raskob, Siamak Mehrkanoon, Stephan Bialonski

benchmarks

arXiv AI

Jul 15

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

arXiv:2603. 28583v2 Announce Type: replace-cross Abstract: Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations.

By Yanjie Zhang, Yafei Li, Rui Sheng, Zixin Chen, Yanna Lin, Huamin Qu, Lei Chen, Yushi Sun

llms agents computer-vision nlp fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 15

Enabling Energy-Efficient Simultaneous Multi-Task Reinforcement Learning through Spiking Neural Networks with Active Dendrites for Bio-inspired Generalist Agents

arXiv:2412. 04847v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has demonstrated remarkable capabilities in training agents to solve complex tasks autonomously, such as mobile robots, UAVs/UGVs, and game-playing agents).

By Rachmad Vidya Wicaksana Putra, Avaneesh Devkota, Muhammad Shafique

agents reinforcement-learning robotics benchmarks

arXiv AI

Jul 15

Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers

arXiv:2507. 15833v3 Announce Type: replace-cross Abstract: Human vision is a highly active process driven by gaze, which directs attention to task-relevant regions through foveation, dramatically reducing visual processing.

By Ian Chuang, Jinyu Zou, Andrew Lee, Dechen Gao, Iman Soltani

llms computer-vision nlp robotics benchmarks safety

arXiv Machine Learning

Jul 15

VQCSim: When Does Compile-Once Statevector Simulation Beat Generic Quantum Frameworks?

arXiv:2607. 11985v1 Announce Type: cross Abstract: Hybrid quantum-classical machine learning workflows repeatedly evaluate many small parametrized circuits during training and model exploration.

By Anton Firc, Martin Pere\v{s}\'ini, Vojt\v{e}ch Mr\'azek, Kamil Malinka, Vojt\v{e}ch Stan\v{e}k, Zbyn\v{e}k Li\v{c}ka, Nouhaila Innan, Walid El Maouaki, Alberto Marchisio, Muhammad Shafique

agents benchmarks

arXiv Machine Learning

Jul 15

Hyperellipsoid Density Sampling: Exploitative Sequences to Accelerate High-Dimensional Numerical Optimization

arXiv:2511. 07836v5 Announce Type: replace-cross Abstract: The curse of dimensionality remains a persistent challenge in modern optimization problems.

By Julian G. Soltes

benchmarks

arXiv AI

Jul 15

How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks

arXiv:2607. 12338v1 Announce Type: new Abstract: Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting.

By Wei-Jung Huang

llms agents benchmarks

arXiv AI

Jul 15

Xray-Visual Models: Scaling Vision models on Industry Scale Data

arXiv:2602. 16918v2 Announce Type: replace-cross Abstract: We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data.

By Shlok Mishra, Tsung-Yu Lin, Linda Wang, Hongli Xu, Yimin Liu, Michael Hsu, Chaitanya Ahuja, Hao Yuan, Jianpeng Cheng, Hong-You Chen, Haoyuan Xu, Chao Li, Sreya Dutta Roy, Abhijeet Awasthi, Jihye Moon, Don Husa, Michael Ge, Sumedha Singla, Arkabandhu Chowdhury, Phong Dingh, Satya Narayan Shukla, Yonghuan Yang, David Jacobs, Qi Guo, Jun Xiao, Xiangjun Fan, Aashu Singh

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 15

SeqGPT: A Constrained Transformer Agent for the Inverse Designof Multi-Panel Composite Structures

arXiv:2607. 11910v1 Announce Type: cross Abstract: Optimizing composite stacking sequences to match continuous targets (e.

By Driss Chraibi (Toulouse INP), Alejandro Garc\'ia Pis (IRIT), St\'ephane Grihon (IRIT), Sixin Zhang (IRIT)

llms agents benchmarks

arXiv AI

Jul 15

IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment

arXiv:2607. 12375v1 Announce Type: cross Abstract: Image Quality Assessment (IQA) in open-world environments remains challenging due to limited generalization and interpretability.

By Jinjian Wu, Jiaqi Tang, Wei Wei, Yingying Yan, Jianmin Chen, Botong Geng, Lei Zhang, Qifeng Chen

llms multimodal benchmarks safety

Get the daily briefing