Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv Machine Learning

Jun 16

A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT

arXiv:2606. 16991v1 Announce Type: cross Abstract: Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload.

By Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel

multimodal benchmarks

arXiv Machine Learning

Jun 16

PHINN: Persistent Homology Inspired Neural Network for Rare-Event Time Series Generation

arXiv:2606. 15452v1 Announce Type: new Abstract: Rare events in time series are critical to model but hard to learn due to data scarcity.

By Emre Yusuf, Ren Takahashi, Jayabrata Bhaduri

rag diffusion multimodal benchmarks safety

arXiv AI

Jun 16

QoS-Aware Token Scheduling and Private Data Valuation for Multi-Modal Agentic Networks

arXiv:2606. 15573v1 Announce Type: new Abstract: In agentic systems, human-generated data records anchor the value of AI services.

By Yao Du, Jing Liu, Pengfei Xu, Zehua Wang, Victor C. M. Leung, Cyril Leung, Victoria Lemieux

agents multimodal benchmarks safety

arXiv AI

Jun 16

PISA: A Pragmatic Psych-Inspired Unified Memory System for Enhanced AI Agency

arXiv:2510. 15966v2 Announce Type: replace Abstract: Memory systems are fundamental to AI agents, yet existing work often lacks adaptability to diverse tasks and overlooks the constructive and task-oriented role of AI agent memory.

By Shian Jia, Ziyang Huang, Xinbo Wang, Haofei Zhang, Mingli Song

agents benchmarks

arXiv AI

Jun 16

Uncertainty Quality of VGGT: An Analysis on the DTU Benchmark Dataset

arXiv:2606. 16479v1 Announce Type: cross Abstract: Visual Geometry Grounded Transformer (VGGT) has already attracted a great deal of attention in a short period of time, not least due to the Best Paper Award at CVPR-2025.

By Markus Hillemann, Robert Langend\"orfer, Steven Landgraf, Markus Ulrich

llms benchmarks

arXiv Machine Learning

Jun 16

From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation

arXiv:2606. 14791v1 Announce Type: cross Abstract: Self-supervised learning advances audio representation for multimedia analysis.

By Fengrui Liu, Ruiyang Huang, Qijian Zheng, Yuanfang Wang, Feng Liu

llms diffusion benchmarks safety

arXiv AI

Jun 16

Probing Low Frame Rate Degradation in Neural Audio Codecs

arXiv:2606. 16969v1 Announce Type: cross Abstract: Low frame rates in neural audio codecs are attractive for autoregressive speech synthesis, where the generation cost scales linearly with the sequence length.

By Alex Gichamba, Moise Busogi

benchmarks

arXiv AI

Jun 16

Visualizing Uncertainty: Spatial Maps of Missing and Conflicting Evidence in Deep Learning

arXiv:2606. 15767v1 Announce Type: cross Abstract: Understanding when and why deep neural networks are uncertain is crucial for deploying reliable machine learning systems in safety-critical domains.

By Dong Hyun Jeong, Feng Chen, Jin-Hee Cho, Lance M. Kaplan, Audun J{\o}sang, Soo-Yeon Ji

benchmarks safety

arXiv Machine Learning

Jun 16

REFLEX: Reflective Evolution from LLM Experience

arXiv:2606. 16496v1 Announce Type: cross Abstract: Large multimodal language models (LLMs) have emerged as powerful tools for guiding evolutionary search toward interpretable programmatic policies.

By Pan Wang

llms multimodal benchmarks

arXiv AI

Jun 16

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.

By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 16

Theorem-Grounded Execution Ontologies for Interpretable Machine Reasoning

arXiv:2606. 16010v1 Announce Type: cross Abstract: Large language models have achieved impressive performance on reasoning tasks spanning mathematics, science, programming, and commonsense inference.

By Raghu Anantharangachar

llms benchmarks

arXiv AI

Jun 16

When Generator Replay Degrades: Projected Rehearsal Orchestration for Heterogeneous Federated Class-Incremental Learning

arXiv:2606. 15695v1 Announce Type: cross Abstract: Federated class-incremental learning (FCIL) becomes substantially harder when clients observe different label subsets, progress through tasks at different stages, and provide uneven supervision for the same semantic concepts.

By Thinh T. H. Nguyen, Khoa D. Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong

agents benchmarks safety

arXiv Machine Learning

Jun 16

Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense

arXiv:2605. 30837v2 Announce Type: replace-cross Abstract: Prompt-injection detectors are heterogeneous: each is strong on a different slice of attacks, and none is always reliable.

By Shuhao Zhang, Jiarui Li, Qi Cao, Ruiyi Zhang, Pengtao Xie

llms agents benchmarks safety

arXiv Machine Learning

Jun 16

Lesion-DDPM: Lesion-Enhanced 3D Diffusion for MS MRI Synthesis

arXiv:2606. 15457v1 Announce Type: cross Abstract: 3D FLAIR MRI is widely recommended as one of the standard MRI sequences for brain imaging in multiple sclerosis (MS), but publicly available MS datasets remain relatively small and vary across scanners, acquisition protocols, and lesion patterns.

By Weidong Zhang, Yongchan Jung, Shafayat Mowla Anik, Furen Xiao, Vasudevan Janarthanan, Enkhzaya Chuluunbaatar, Byeong Kil Lee, Jeeho Ryoo

diffusion computer-vision benchmarks

arXiv AI

Jun 16

RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking

arXiv:2604. 20623v2 Announce Type: replace-cross Abstract: Traditional change detection identifies where changes occur, but does not explain what changed in natural language.

By Roie Kazoom, Yotam Gigi, George Leifman, Tomer Shekel, Genady Beryozkin

rag computer-vision multimodal benchmarks

arXiv AI

Jun 16

AgentFairBench: Do LLM Agents Discriminate When They Act?

arXiv:2606. 16723v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers.

By Triveni Morla, Rohith Reddy Bellibaltu, Manpreet Singh, Manmeet Singh Kapoor

llms agents benchmarks safety

arXiv Machine Learning

Jun 16

Formalizing and Mitigating Structural Distortion in LLM Attention for Zero-Shot Graph Reasoning

arXiv:2606. 15633v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs).

By Donald Loveland, Puja Trivedi, Ari Weinstein, Edward W Huang, Danai Koutra

llms rag benchmarks

arXiv AI

Jun 16

ScoutVLA: UAV-Centric Active Perception via a Dual-Expert VLA Model for Open-World Embodied Question Answering

arXiv:2606. 14772v1 Announce Type: cross Abstract: Aerial Embodied Question Answering (EQA) requires Unmanned Aerial Vehicles (UAVs) to actively perceive the environment and answer natural language questions.

By Wenhao Lu, Zhengqiu Zhu, Xiaofeng Wang, Xiaoran Zhang, Yatai Ji, Yong Zhao, Yue Hu, Yingzhen Nie, Jinlong Zhu, Zheng Zhu

diffusion nlp robotics multimodal benchmarks

arXiv AI

Jun 16

Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localization

arXiv:2606. 16898v1 Announce Type: cross Abstract: Detecting unanswerable user queries remains essential for the reliable deployment of real-world embodied agents.

By Dongbin Na, Chanwoo Kim, Giyun Choi, Dooyoung Hong

llms agents nlp robotics multimodal benchmarks

arXiv AI

Jun 16

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

arXiv:2606. 16825v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory.

By Martin Jaggi

llms efficiency benchmarks

Get the daily briefing