Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv Machine Learning

Jun 2

Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting

arXiv:2512. 22702v2 Announce Type: replace Abstract: Deep learning models have grown popular in time series applications.

By Valentina Moretti, Ivan Marisca, Cesare Alippi, Andrea Cini

benchmarks

arXiv AI

Jun 2

Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization

arXiv:2510. 10982v2 Announce Type: replace-cross Abstract: Recent AI regulations increasingly emphasize the need for mechanisms that preserve the utility of data for AI innovation while preventing misuse, particularly by enforcing purpose limitation in downstream AI applications.

By Zihan Wang, Zhiyong Ma, Zhongkui Ma, Shuofeng Liu, Akide Liu, Derui Wang, Minhui Xue, Guangdong Bai

llms multimodal benchmarks safety

arXiv AI

Jun 2

Enhancing BiGRU with a KAN Block for Legal Document Classification and Summarization

arXiv:2606. 00116v1 Announce Type: cross Abstract: This study introduces a novel architecture of KAN-based BiGRU model for the task of classification and summarization of legal documents in a low-resource multilingual setup.

By Ahmed Faizul Haque Dhrubo, Souvik Pramanik, Most. Aysha Siddika Sumona, Shahnewaz Siddique, Mohammad Ashrafuzzaman Khan, Mohammad Abdul Qayum, Mohsin Sajjad

llms nlp benchmarks

arXiv AI

Jun 2

FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

arXiv:2604. 03893v2 Announce Type: replace Abstract: Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference.

By Zeyu Wang, Jingye Xu, Xiaogang Li, Peiyao Xiao, Qinhao Kong, Ben Wang, Chengliang Xu, Zichao Chen, Bing Zhao, Hu Wei

llms multimodal benchmarks

arXiv AI

Jun 2

Transferring Information Across Interventions in Causal Bayesian Optimization

arXiv:2606. 01457v1 Announce Type: new Abstract: Bayesian optimization is a popular way to optimize expensive systems, where every experiment, simulation, or intervention costs time or money.

By Mohammad Ali Javidian

benchmarks

arXiv AI

Jun 2

Recognize Your Orchestrator: An Entropy Dynamics Perspective for LLM Multi-Agent Systems

arXiv:2606. 01351v1 Announce Type: new Abstract: The transition from single-turn models to Multi-Agent Systems (MAS) promises enhanced problem-solving capabilities, yet the centralized orchestration topology remains a critical point of fragility.

By Junze Zhu, Weihao Chen, Xuanwang Zhang, Zhen Wu, Xinyu Dai

llms agents benchmarks

arXiv AI

Jun 2

Unsupervised Cognition

arXiv:2409. 18624v4 Announce Type: replace Abstract: Unsupervised learning methods have a soft inspiration in cognition models.

By Alfredo Ibias, Hector Antona, Guillem Ramirez-Miranda, Enric Guinovart, Eduard Alarcon

benchmarks

arXiv AI

Jun 2

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

arXiv:2606. 00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment.

By Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

llms multimodal benchmarks safety

arXiv AI

Jun 2

MViewRouter: Internalizing Geometric Equivariance via Multi-view Alternating Attention for Combinatorial Routing

arXiv:2606. 01084v1 Announce Type: cross Abstract: Combinatorial routing problems such as the Traveling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP) are fundamental NP-hard problems with broad real-world applications.

By Shiyan Liu, Bohan Tan, Yaoxin Wu, Yan Jin

reinforcement-learning benchmarks safety

arXiv AI

Jun 2

STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing

arXiv:2606. 01126v1 Announce Type: cross Abstract: Pruning is a process designed to reduce the number of weights in a large neural network.

By Shir Maon, Odelia Melamed, Adi Shamir

efficiency benchmarks

arXiv AI

Jun 2

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

arXiv:2606. 01850v1 Announce Type: new Abstract: Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation.

By Yujia Tong, Yuxi Wang, Yunyang Wan, Tian Zhang, Junhao Dong, Jingling Yuan

llms nlp efficiency benchmarks safety

arXiv AI

Jun 2

The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

arXiv:2605. 05427v2 Announce Type: replace Abstract: Refusal rates are a poor proxy for LLM safety, i.

By Alif Al Hasan, Sumon Biswas

llms benchmarks safety

arXiv AI

Jun 2

InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning

arXiv:2509. 12263v3 Announce Type: replace Abstract: Large multimodal models (LMMs) encode physical laws observed during training, such as momentum conservation, as parametric knowledge.

By Gautam Sreekumar, Vishnu Naresh Boddeti

agents nlp multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Beyond the Simplex: Balanced Prototype Geometry for Scorer-Agnostic Open-Set Recognition

arXiv:2606. 01883v1 Announce Type: new Abstract: Open-set recognition (OSR) requires a classifier to reject inputs from unseen classes which is essential in safety-critical settings such as medical imaging.

By Mayank Sharma, Rohit Kumar Mourya

rag benchmarks safety

arXiv Machine Learning

Jun 2

Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging

arXiv:2606. 02339v1 Announce Type: new Abstract: Entropy minimization (EM) is the dominant objective for test-time adaptation, yet its failure mode, model collapse, remains poorly understood.

By Tim Nielen, Sameer Ambekar, Johannes Kiechle, Daniel M. Lang, Julia A. Schnabel

benchmarks safety

arXiv Machine Learning

Jun 2

Semi-Supervised Hyperbolic Hierarchical Clustering with Set-Level Structural Priors

arXiv:2606. 01525v1 Announce Type: new Abstract: Semi-supervised hierarchical clustering aims to learn a tree structure consistent with data patterns and user-provided supervision.

By Junjing Zheng, Xinyu Zhang, Xiangfeng Qiu, Chengliang Song, Weidong Jiang

rag benchmarks

arXiv Machine Learning

Jun 2

A Per-Component Diagnostic Protocol for Neural HJB-PIDE Solvers under Control-Dependent L\'evy Jumps

arXiv:2606. 01122v1 Announce Type: new Abstract: We propose a five-step diagnostic protocol for residual-trained neural HJB-PIDE solvers with control-dependent L\'evy jumps, targeting a general failure mode of neural PDE methods: a learned solution can match headline scalar diagnostics while miscomputing an operator inside its training loss.

By R. Drissi

diffusion benchmarks

arXiv Machine Learning

Jun 2

MedGym:A Unified Continuous-Time Benchmark for Dynamic Medical Treatment Reinforcement Learning

arXiv:2606. 01028v1 Announce Type: new Abstract: Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements and interventions are performed at irregular intervals, and treatment effects vary substantially across individuals.

By Yuepeng Wang, Ken Kawano, Yongqi Zhou, Yoshihiko Fujisawa, Richard Weiss, Akifumi Wachi, Katsuki Fujisawa, Ying Chen, Mehrshad Sadria, Xin Liu, Kyoung-Sook Kim, Xiao Hu, Sebastien Gros, Xun Shen

reinforcement-learning benchmarks safety

arXiv AI

Jun 2

Why Self-Inconsistency Arises in GNN Explanations and How to Exploit It

arXiv:2605. 07527v2 Announce Type: replace-cross Abstract: Recent work has observed that explanations produced by Self-Interpretable Graph Neural Networks (SI-GNNs) can be self-inconsistent: when the model is reapplied to its own explanatory graph subset, it may produce a different explanation.

By Wenxin Tai, Yaqian Liu, Ting Zhong, Fan Zhou

benchmarks

arXiv AI

Jun 2

T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models

arXiv:2504. 04718v2 Announce Type: replace-cross Abstract: Recent studies have demonstrated that test-time compute scaling effectively improves the performance of small language models (sLMs).

By Minki Kang, Jongwon Jeong, Jaewoong Cho

llms efficiency benchmarks

Get the daily briefing