Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,452 stories · RSS feed

arXiv Machine Learning

Jun 19

Structure-Oriented Randomized Neural Networks for Poisson-Nernst-Planck and Poisson-Nernst-Planck-Navier-Stokes Systems

arXiv:2606. 19912v1 Announce Type: cross Abstract: We develop a structure-oriented randomized neural network framework, termed SO-RaNN, for the Poisson-Nernst-Planck (PNP) system and the Poisson-Nernst-Planck-Navier-Stokes (PNP-NS) system.

By Yunlong Li, Fei Wang

benchmarks

arXiv Machine Learning

Jun 19

Bioacoustic Geolocation: Species Sounds as Geographic Signals

arXiv:2505. 18726v3 Announce Type: replace-cross Abstract: Can we determine someone's geographic location solely from the sounds they hear?

By Mustafa Chasmai, Wuao Liu, Subhransu Maji, Grant Van Horn

multimodal benchmarks

arXiv AI

Jun 19

Speeding up the annotation process in semantic segmentation industrial applications

arXiv:2606. 19934v1 Announce Type: cross Abstract: Current machine learning models commonly require large and well-annotated datasets.

By Marta Fernandez-Moreno, Margarita Guerrero, Rosalia Rementeria, Pablo Mesejo, Raul Moreno

computer-vision benchmarks

arXiv AI

Jun 19

Residual-Space Evolutionary Optimization via Flow-based Generative Models

arXiv:2606. 20084v1 Announce Type: new Abstract: Data editing with generative methods typically requires differentiable objectives and gradient-based search.

By Zhuo Cao, Lena Krieger, Fernanda Nader, Xuan Zhao, Hanno Scharr, Ira Assent

diffusion benchmarks safety

arXiv Machine Learning

Jun 19

EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning

arXiv:2510. 19893v2 Announce Type: replace Abstract: Medical AI systems demonstrated impressive diagnostic performance, yet they routinely show uneven accuracy across demographic groups, disadvantaging underrepresented populations.

By Shiqi Dai, Wei Dai, Jiaee Cheong, Paul Pu Liang

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 19

Evaluation of EEG Foundation Models for Event-Based Burst-Suppression Detection in ICU

arXiv:2606. 20074v1 Announce Type: cross Abstract: Burst suppression (BS) is a clinically relevant electroencephalographic (EEG) pattern used to monitor sedation depth and brain activity in critically ill patients, particularly during induced coma in Intensive Care Units (ICUs).

By Elisa Vasta, Thorir Mar Ingolfsson, Andrea Cossettini, Luca Benini, Tilman Beck, Emanuela Keller, Una Pale

fine-tuning benchmarks

arXiv AI

Jun 19

Human Universal Grasping

arXiv:2606. 17054v1 Announce Type: cross Abstract: Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality.

By Kevin Yuanbo Wu, Tianxing Zhou, Isaac Tu, Billy Yan, Irmak Guzey, David Fouhey, Dandan Shan, Lerrel Pinto

robotics benchmarks

arXiv AI

Jun 19

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

arXiv:2606. 19787v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear.

By Jiajun Li, Mingshu Cai, Yixuan Li, Yu Ding, Ran Hou, Guanyu Nie, Xiongwei Han, Wanyuan Wang

llms agents benchmarks

arXiv AI

Jun 19

Benchmarking Agentic Review Systems

arXiv:2606. 19749v1 Announce Type: new Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.

By Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan

llms agents benchmarks

arXiv AI

Jun 19

Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks

arXiv:2606. 19489v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space.

By Ya Wang, Adrian Paschke

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

arXiv:2606. 20118v1 Announce Type: cross Abstract: Vision-language-action (VLA) policies have shown strong potential for general-purpose manipulation, yet they often fail on novel, out-of-distribution objects whose appearance or geometry deviates from the training distribution.

By Jonghoon Lee, Seong Hyeon Park, Byungwoo Jeon, Minha Lee, Jinwoo Shin

robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 19

Frequency-Aware Flow Matching for Continuous and Consistent Robotic Action Generation

arXiv:2606. 20135v1 Announce Type: cross Abstract: Flow matching has emerged as a standard paradigm for robotic manipulation owing to its strong expressive power for modelling complex, multimodal action distributions, alongside similar approaches like diffusion policy.

By Jianing Guo, Fangzheng Chen, Zihao Mao, Wong Lik Hang Kenny, Zhenhong Wu, Yu Li, Yishuai Cai, Yuanpei Chen, Yikun Ban, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Huijie Zhao, Simin Li

diffusion robotics multimodal benchmarks safety

arXiv AI

Jun 19

Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

arXiv:2606. 20470v1 Announce Type: cross Abstract: Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents.

By Reza Soosahabi, Vivek Namsani

agents multimodal benchmarks safety

arXiv Machine Learning

Jun 19

On the Redundancy of Timestep Embeddings in Diffusion Models

arXiv:2606. 20416v1 Announce Type: new Abstract: Diffusion models rely heavily on explicit timestep embeddings to modulate the denoising process across various noise scales.

By Jos\'e A. Ch\'avez

llms rag diffusion benchmarks

arXiv Machine Learning

Jun 19

Shifting-based Optimizable Linear Relaxations for General Activation Functions

arXiv:2606. 20292v1 Announce Type: new Abstract: The use of neural networks (NNs) is rapidly increasing, including in safety- and security-critical domains.

By Philipp Kern, L\'aszl\'o Antal, Erika \'Abr\'aham, Carsten Sinz

benchmarks safety

arXiv AI

Jun 19

MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environments

arXiv:2606. 19893v1 Announce Type: new Abstract: Deep research agents have demonstrated remarkable capabilities in autonomous information gathering and synthesis, yet their training remains constrained by the static nature of simulated environments, the limits of fact-retrieval-only task designs, and the inefficiency of outcome-based reinforcement learning.

By Wei Yu, Suxing Liu, Minjie Yu, Jiahao Wang, Zhijian Zheng, Haocheng Deng, Bing Li

agents reinforcement-learning benchmarks safety

arXiv AI

Jun 19

Diffusion Language Models: An Experimental Analysis

arXiv:2606. 19475v1 Announce Type: new Abstract: Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks.

By Thomas Bertolani, Davide Bucciarelli, Leonardo Zini, Marcella Cornia, Lorenzo Baraldi

llms diffusion benchmarks

arXiv AI

Jun 19

Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer

arXiv:2606. 19346v1 Announce Type: cross Abstract: We study cross-lingual transfer by fine-tuning seven large language models (4B--671B parameters) on Arabic and evaluating zero-shot reading comprehension on Semitic languages and non-Semitic controls.

By Ahmed Haj Ahmed, Ruochen Zhang, Alvin Grissom II

llms fine-tuning benchmarks safety

arXiv AI

Jun 19

FlowFake: Liquid Networks for Audio Deepfake Detection

arXiv:2606. 19579v1 Announce Type: cross Abstract: Audio deepfakes generated by neural text-to-speech and voice-cloning systems threaten speaker verification and public discourse at scale.

By Shivaay Dhondiyal, Divyansh Sharma, Dinesh Kumar Vishwakarma

multimodal benchmarks

arXiv AI

Jun 19

LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems

arXiv:2606. 20408v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized.

By Hanwool Lee, Dasol Choi, Bokyeong Kim, Seung Geun Kim, Haon Park

llms agents benchmarks safety

Get the daily briefing