Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv Machine Learning

Jun 4

veriFIRE: an Industrial Case Study in Verifying Consistency Properties for a DNN-Based Wildfire Detection System

arXiv:2606. 04121v1 Announce Type: cross Abstract: We present our ongoing work on the veriFIRE project: a collaboration between industry and academia, aimed at applying verification to increase the reliability of a real-world, safety-critical system.

By Idan Refaeli, Maya Swisa, Itay Buchnik, Alon Zada, Guy Amir, Elad Mandelbaum, Ziv Freund, Guy Katz

benchmarks safety

arXiv AI

Jun 4

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

arXiv:2606. 04023v1 Announce Type: cross Abstract: While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.

By Jie Li, Wenzhao Wu, Junqi Hu, Qinrui Zheng, Bowen Wu, Juepeng Zheng, Yutong Lu, Haohuan Fu

llms benchmarks

arXiv Machine Learning

Jun 4

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

arXiv:2603. 09803v2 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance.

By Tiehua Mei, Minxuan Lv, Leiyu Pan, Zhenpeng Su, Hongru Hou, Hengrui Chen, Ao Xu, Deqing Yang

llms reinforcement-learning benchmarks

arXiv AI

Jun 4

Characterizing initial human-AI proof formalization workflows

arXiv:2606. 04273v1 Announce Type: new Abstract: For centuries, human mathematicians have written proofs to substantiate their mathematical arguments; yet, the ability to automatically verify the validity of proofs has long been a challenge.

By Katherine M. Collins, Simon Frieder, Jonas Bayer, Jacob Loader, Jeck Lim, Peiyang Song, Fabian Zaiser, Lexin Zhou, Shanda Li, Sam Looi, Joshua B. Tenenbaum, Umang Bhatt, Adrian Weller, Jose Hernandez-Orallo, Cameron E. Freer, Valerie Chen, Ilia Sucholutsky

benchmarks

arXiv AI

Jun 4

RUBAS: Rubric-Based Reinforcement Learning for Agent Safety

arXiv:2606. 04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation.

By Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 4

Bayesian Membership Privacy for Graph Neural Networks

arXiv:2606. 04069v1 Announce Type: cross Abstract: Existing privacy analyses for Graph Neural Networks (GNNs) largely inherit assumptions from non-graph settings, overlooking structural correlations and stochastic training-graph sampling.

By Sinan Y{\i}ld{\i}r{\i}m, Megha Khosla

benchmarks safety

arXiv Machine Learning

Jun 4

CADET: A Modular Platform for Evaluating Distributed Cooperative Autonomy in Connected Autonomous Vehicles

arXiv:2606. 04072v1 Announce Type: cross Abstract: Deep learning models are increasingly central to autonomous vehicle (AV) pipelines, yet their integration has traditionally followed a monolithic design where perception, planning, and control execute on a single onboard computer.

By Pragya Sharma, Brian Wang, Mani Srivastava

agents benchmarks safety

arXiv AI

Jun 4

DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities

arXiv:2606. 04205v1 Announce Type: cross Abstract: The growing popularity and capacity of generative models have eroded the distinction between human and machine-generated content, motivating a growing body of work on detection across text, images, and audio.

By Sajad Ebrahimi, Nima Jamali, Bardia Shirsalimian, Kelly McConvey, Wentao Zhang, Jalehsadat Mahdavimoghaddam, Maksym Taranukhin, Maura Grossman, Vered Shwartz, Yuntian Deng, Ebrahim Bagheri

multimodal benchmarks

arXiv AI

Jun 4

Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty

arXiv:2606. 04342v1 Announce Type: cross Abstract: Multi-step time series forecasting (MSF) is commonly evaluated using point-wise error metrics such as mean squared error (MSE), implicitly treating the conditional mean as a sufficient target.

By Riku Green, Zahraa S. Abdallah, Telmo M Silva Filho

benchmarks

arXiv Machine Learning

Jun 4

Efficient and Training-Free Single-Image Diffusion Models

arXiv:2606. 04299v1 Announce Type: cross Abstract: We consider the problem of generating images whose internal structure -- defined by the distribution of patches across multiple scales -- matches that of a single reference image.

By Haojun Qiu, Kiriakos N. Kutulakos, David B. Lindell

diffusion benchmarks

arXiv AI

Jun 4

Rethinking Sales Lead Scoring with LLM-based Hierarchical Preference Ranking

arXiv:2606. 04387v1 Announce Type: cross Abstract: Sales lead conversion in high-stakes domains (e.

By Chenyu Zhang, Yiwen Liu, Yin Sun, Xinyuan Zhang, Yuji Cao, Junming Jiao, Juyi Qiao

llms benchmarks safety

arXiv Machine Learning

Jun 4

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

arXiv:2606. 04434v1 Announce Type: cross Abstract: Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks.

By Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 4

Neural Variance-aware Dueling Bandits with Deep Representation and Shallow Exploration

arXiv:2506. 01250v3 Announce Type: replace Abstract: We introduce the first variance-aware algorithms for contextual dueling bandits that leverage shallow exploration strategies with neural networks for nonlinear utility approximation.

By Youngmin Oh, Jinje Park, Taejin Paik

benchmarks

arXiv Machine Learning

Jun 4

Identifying and Correcting Label Noise for Robust GNNs via Influence Contradiction

arXiv:2601. 17469v2 Announce Type: replace Abstract: Graph Neural Networks (GNNs) have shown remarkable capabilities in learning from graph-structured data with various applications such as social analysis and bioinformatics.

By Wei Ju, Wei Zhang, Siyu Yi, Zhengyang Mao, Yifan Wang, Jingyang Yuan, Zhiping Xiao, Ziyue Qiao, Ming Zhang

diffusion benchmarks

arXiv Machine Learning

Jun 4

On the Expressive Power of Permutation-Equivariant Weight-Space Networks

arXiv:2602. 01083v2 Announce Type: replace Abstract: Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks.

By Adir Dayan, Yam Eitan, Haggai Maron

benchmarks

arXiv Machine Learning

Jun 4

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.

By Jianlu Shen, Fu Feng, Yucheng Xie, Jiaqi Lv, Xin Geng

fine-tuning benchmarks

arXiv Machine Learning

Jun 4

Towards interpretable AI with quantum annealing feature selection

arXiv:2604. 25649v2 Announce Type: replace Abstract: Deep learning models are used in critical applications, in which mistakes can have serious consequences.

By Francesco Aldo Venturelli, Emanuele Costa, Sikha O K, Bruno Juli\'a-D\'iaz, Miguel A. Gonz\'alez Ballester, Alba Cervera-Lierta

computer-vision benchmarks

arXiv Machine Learning

Jun 4

The Perception-Physics Paradox: Probing Scientific Alignment with TC-Bench

arXiv:2605. 24782v2 Announce Type: replace Abstract: While Vision Foundation Models (VFMs) excel at predictive tasks on satellite imagery, their performance can arise from visual correlations rather than underlying structural invariants, making even perception-based out-of-distribution accuracy a poor proxy for scientific utility.

By Dingling Yao, Andrea Polesello, Adeel Pervez, Caroline Muller, Francesco Locatello

benchmarks safety

arXiv AI

Jun 4

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

arXiv:2601. 09719v3 Announce Type: replace-cross Abstract: Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning.

By Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, Kyungwoo Song

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 4

BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation format

arXiv:2509. 02655v3 Announce Type: replace-cross Abstract: Many AI alignment discussions of "runaway optimisation" focus on RL agents: unbounded utility maximisers that over-optimise a proxy objective (e.

By Roland Pihlakas (for the Three Laws collaboration), Sruthi Susan Kuriakose (for the Three Laws collaboration)

llms agents reinforcement-learning benchmarks safety

Get the daily briefing