Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv AI

Jun 30

Beyond Drug Discovery: The Nanotechnology Molecular Optimization (NMO) Benchmark

arXiv:2606. 30170v1 Announce Type: cross Abstract: Generative molecular design is shaped by simple proxy benchmarks for drug-like properties and models pretrained on large pharmaceutical datasets.

By Matthias Blaschke, Daniel Kienzle, Zsuzsanna Koczor-Benda, Julian Lorenz, Rainer Lienhart, Fabian Pauly

benchmarks safety

arXiv AI

Jun 30

Physically-Constrained Harmonic Separation for Robust Heart and Respiratory Rate Estimation from Wrist Photoplethysmography

arXiv:2606. 30156v1 Announce Type: cross Abstract: Wrist-worn photoplethysmography (PPG) enables continuous monitoring of cardiopulmonary physiology, but reliable heart rate (HR) and respiratory rate (RR) estimation in free-living conditions remains challenging due to non-stationary motion artifacts that spectrally overlap with physiological dynamics.

By Nouhaila Fraihi, Ouassim Karrakchou, Mounir Ghogho

benchmarks safety

arXiv AI

Jun 30

SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings

arXiv:2606. 28465v1 Announce Type: cross Abstract: This work examines perturbation generalization in spatial foundation-model embeddings derived from fluorescence microscopy images.

By Jake Y. Chen, Huu Phong Nguyen, Fuad Al Abir, Ehsan Saghapour

rag benchmarks

arXiv AI

Jun 30

ProSpec RL: Plan Ahead, then Execute

arXiv:2407. 21359v2 Announce Type: replace-cross Abstract: Imagining potential outcomes of actions before execution helps agents make more informed decisions, a prospective thinking ability fundamental to human cognition.

By Liangliang Liu, Yi Guan, BoRan Wang, Rujia Shen, Yi Lin, Chaoran Kong, Lian Yan, Jingchi Jiang

agents reinforcement-learning benchmarks

arXiv AI

Jun 30

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

arXiv:2603. 29139v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks.

By Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

llms agents multimodal benchmarks

arXiv AI

Jun 30

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605. 30295v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited.

By Valentina Bui Muti, Eug\'enie Dulout, Ziquan Fu

llms benchmarks

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization

arXiv:2606. 29951v1 Announce Type: new Abstract: Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models.

By Hugo L. Hammer, Vajira Thambawita, Kristoffer Herland Hellton, P{\aa}l Halvorsen

benchmarks safety

arXiv AI

Jun 30

TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding

arXiv:2508. 07683v2 Announce Type: replace-cross Abstract: Video Temporal Grounding (VTG) aims to localize specific video segments corresponding to natural language queries.

By Chaohong Guo, Xun Mo, Yongwei Nie, Fei Ma, Xuemiao Xu, Chengjiang Long

llms agents reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 30

Resonant Brane Splatting for Arbitrary-Scale Super-Resolution

arXiv:2606. 29453v1 Announce Type: cross Abstract: Arbitrary-Scale Super-Resolution (ASR) reconstructs images at continuous magnification factors.

By Giulio Federico, Giuseppe Amato, Claudio Gennaro, Fabio Carrara, Marco Di Benedetto

multimodal benchmarks

arXiv AI

Jun 30

Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving

arXiv:2606. 30537v1 Announce Type: cross Abstract: Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations.

By Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong

agents robotics fine-tuning benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 30

Warm-Starting Iterative Gaussian Processes for Faster Sequential Inference

arXiv:2511. 16340v2 Announce Type: replace Abstract: Efficient Gaussian process (GP) inference is critical for sequential decision-making tasks such as active learning, online prediction, and Bayesian optimization.

By Alan Yufei Dong, Jihao Andreas Lin, Jos\'e Miguel Hern\'andez-Lobato

benchmarks

arXiv AI

Jun 30

Temporal Feature Extractors in EEG Foundation Models: A Controlled Comparison Including a Pretrained Time-Series Model

arXiv:2606. 30104v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models aim to learn generalizable representations from large-scale brain recordings.

By Ay\c{s}e Bet\"ul Y\"uce, Chris Joey Leffler, Sarun Varghese, Myra Spiliopoulou, Sebastian Stober

benchmarks

arXiv Machine Learning

Jun 30

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

arXiv:2606. 29788v1 Announce Type: new Abstract: When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.

By Kuan Wang, Chao Zhang

agents multimodal benchmarks

arXiv Machine Learning

Jun 30

KrishokChat: A Citation-Grounded Dataset and Benchmark for Bengali Agricultural Advisory

arXiv:2606. 29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings.

By Khan Raiyan Ibne Reza, Omar Ibne Shahid

rag fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606. 30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge.

By Beatrix Koltai, Gergely Acs, Andras Gazdag

benchmarks

arXiv Machine Learning

Jun 30

Nonlinear mixture model motivated subspace clustering

arXiv:2606. 29261v1 Announce Type: new Abstract: We derive the linear union-of-subspaces (UoS) model for subspace clustering (SC) from the nonlinear mixture model (NMM) used in blind source separation (BSS) to represent a D-dimensional observation vector as an unknown multivariate nonlinear mapping of C latent variables.

By Ivica Kopriva

efficiency benchmarks

arXiv AI

Jun 30

Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation

arXiv:2511. 05852v4 Announce Type: replace-cross Abstract: Knowledge editing (KE) offers a lightweight alternative to retraining for updating large language models (LLMs).

By Yinjie Cheng, Paul Youssef, Christin Seifert, J\"org Schl\"otterer, Zhixue Zhao

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

How to Train Your Long-Context Visual Document Model

arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.

By Austin Veselka

llms nlp fine-tuning benchmarks

Get the daily briefing