Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv Machine Learning

Jul 24

One Round Is All You Need: Analytic Federated Learning for Task-Heterogeneous Multi-Label Medical Image Classification

arXiv:2607. 20641v1 Announce Type: new Abstract: Federated learning (FL) enables multiple clinical institutions to collaboratively train a shared disease classifier without centralizing patient data.

By Afsaneh Mahanipour, Hana Khamfroush

computer-vision benchmarks safety

arXiv Machine Learning

Jul 24

What, Where, and How: Disentangling the Roles of Task, Language, and Model in Code Model Representations

arXiv:2607. 21491v1 Announce Type: cross Abstract: Do independently trained language models come to represent the same thing in the same way?

By Piotr Wilam

llms benchmarks

arXiv AI

Jul 24

Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement

arXiv:2607. 20529v1 Announce Type: cross Abstract: Large Language Model (LLM) ensembles are increasingly used to improve reliability by combining predictions from multiple LLMs.

By Jiawei Zheng, Jiazhen Zhang

llms benchmarks safety

arXiv Machine Learning

Jul 24

Test-Time Scaling via Error Localization

arXiv:2607. 21453v1 Announce Type: new Abstract: Scaling inference-time computation has emerged as a reliable method to improve the performance of large language models on complex reasoning and programming tasks.

By Rajiv Shailesh Chitale, Rahul Madhavan, Taneesh Gupta, Deepanway Ghosal, Aravindan Raghuveer

llms benchmarks

arXiv AI

Jul 24

Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?

arXiv:2607. 20460v1 Announce Type: cross Abstract: Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed.

By Yuzhi Tang, Wentao Ma, Xiling Zhao, Ahmad Salimi, Sepehr Harfi Moridani, Dongming Shen, Jixuan Wang, Abdulrahman Abdulrazzag, Murdock Aubry, Yu-Hua Chen, Daniel Lee, Jaewon Lee, Jonah Mackey, Silin Meng, Nicholas Stranges, Chenxu Xiong, Hao Yu, Yi Zhu, Mu Li, Alex Smola

llms benchmarks

arXiv AI

Jul 24

CLOE: Christoffel Loss Autoencoder for Anomaly Detection

arXiv:2607. 20530v1 Announce Type: cross Abstract: Semi-supervised anomaly detection plays a key role in diverse fields such as process monitoring, healthcare, and finance.

By L\'ea Billet (LAAS, INSA Toulouse, ANITI), Louise Trav\'e-Massuy\`es (LAAS-DISCO, Comue de Toulouse, ANITI), Elodie Chanthery (LAAS), Alexandre Gaffet

diffusion benchmarks

arXiv Machine Learning

Jul 24

Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning

arXiv:2607. 21353v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of specific training data while preserving model utility.

By Billel Habbati, Alessio Merlo, Luca Verderame, Meriem Guerar

efficiency benchmarks

arXiv Machine Learning

Jul 24

CLOAK: Contrastive Guidance for Latent Diffusion-Based Data Obfuscation

arXiv:2512. 12086v2 Announce Type: replace Abstract: Data obfuscation is a promising technique for mitigating attribute inference attacks by semi-trusted parties with access to time-series data emitted by sensors.

By Xin Yang, Omid Ardakanian

diffusion benchmarks safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv AI

Jul 24

ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management

arXiv:2607. 20764v1 Announce Type: new Abstract: We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows.

By Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey

agents benchmarks

arXiv AI

Jul 24

Towards Faithful Graph Explanations with Synergistic Edge Effects via Granular Balls

arXiv:2607. 21381v1 Announce Type: new Abstract: Instance-level explanations aim to reveal the rationale behind a model's decisions for a specific graph.

By Jiancu Chen, Shuyin Xia, Guan Wang, Degang Chen, Fan Chen

benchmarks

arXiv AI

Jul 24

The Active Ingredient in Muon's Grokking

arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.

By Yufeng Wang

rag benchmarks

arXiv AI

Jul 24

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

arXiv:2607. 20988v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving.

By Quanfu Yu, Xian Wu, Hao Xu, Liulong Ma

agents robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

StabilityBench: Benchmarking Instability in LLMs

arXiv:2607. 20558v1 Announce Type: cross Abstract: AI Assistants are increasingly deployed in high-stakes settings, such as healthcare or government services.

By Emma Kondrup, Zachary Yang, Anne Imouza, Reihaneh Rabbany

llms benchmarks safety

arXiv AI

Jul 24

GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning

arXiv:2607. 20730v1 Announce Type: cross Abstract: Large language models increasingly use search tools to retrieve up-to-date information, introducing a new attack surface in which retrieved documents can be manipulated.

By Zhaoqi Wang, Zijian Zhang, Xiaomei Yuan, Pengtao Kou, Jiamou Liu, Zhen Li, Liehuang Zhu

llms benchmarks safety

arXiv AI

Jul 24

Streamliners for Answer Set Programming

arXiv:2604. 19251v2 Announce Type: replace-cross Abstract: Streamliner constraints reduce the search space of combinatorial problems by ruling out portions of the solution space.

By Florentina Voboril (TU Wien), Martin Gebser (University of Klagenfurt), Stefan Szeider (TU Wien), Alice Tarzariol (University of Klagenfurt)

llms benchmarks

arXiv AI

Jul 24

Multimodal Learning for Arcing Detection in Pantograph-Catenary Systems

arXiv:2602. 08792v2 Announce Type: replace-cross Abstract: The pantograph-catenary interface is essential for ensuring uninterrupted and reliable power delivery in electrified rail systems.

By Hao Dong, Eleni Chatzi, Olga Fink

multimodal benchmarks

arXiv Machine Learning

Jul 24

Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion

arXiv:2607. 20742v1 Announce Type: new Abstract: Multimodal learning is a robust approach to improve predictive performance in applications such as medical prognosis.

By Mohammad Raahemi, Ali Sekhavati, Alireza Maleki, Hamid Nasiri

multimodal benchmarks safety data-engineering

arXiv Machine Learning

Jul 24

MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems

arXiv:2509. 22047v3 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available.

By Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Mitsuki Sakamoto, Ryota Mitsuhashi, Eiji Uchibe

nlp reinforcement-learning benchmarks

arXiv Machine Learning

Jul 24

PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics

arXiv:2607. 20653v1 Announce Type: cross Abstract: Predicting how deformable objects evolve under robotic manipulation is a longstanding challenge.

By Haocheng Yin, Shuohan Tao, Yongsheng Chen, Lu Gan

robotics benchmarks

Get the daily briefing