Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv Machine Learning

Jun 4

Towards interpretable AI with quantum annealing feature selection

arXiv:2604. 25649v2 Announce Type: replace Abstract: Deep learning models are used in critical applications, in which mistakes can have serious consequences.

By Francesco Aldo Venturelli, Emanuele Costa, Sikha O K, Bruno Juli\'a-D\'iaz, Miguel A. Gonz\'alez Ballester, Alba Cervera-Lierta

computer-vision benchmarks

arXiv AI

Jun 4

PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

arXiv:2602. 01146v2 Announce Type: replace Abstract: Conversational assistants are increasingly integrating long-term memory with large language models (LLMs).

By Sidharth Pulipaka, Oliver Chen, Manas Sharma, Taaha S Bajwa, Vyas Raina, Ivaxi Sheth

llms benchmarks safety

arXiv Machine Learning

Jun 4

The Perception-Physics Paradox: Probing Scientific Alignment with TC-Bench

arXiv:2605. 24782v2 Announce Type: replace Abstract: While Vision Foundation Models (VFMs) excel at predictive tasks on satellite imagery, their performance can arise from visual correlations rather than underlying structural invariants, making even perception-based out-of-distribution accuracy a poor proxy for scientific utility.

By Dingling Yao, Andrea Polesello, Adeel Pervez, Caroline Muller, Francesco Locatello

benchmarks safety

arXiv Machine Learning

Jun 4

Neural Autoregressive Control Variates for the Quantum Monte Carlo Sign Problem

arXiv:2605. 26814v2 Announce Type: replace-cross Abstract: We train a pair of autoregressive models to construct zero-mean control variates to mitigate the sign problem in quantum Monte Carlo simulations.

By Bei Qiao, Lei Wang

llms benchmarks

arXiv Machine Learning

Jun 4

Structure-Aware Prediction of PROTAC-Mediated Protein Degradability via Graph Neural Networks

arXiv:2606. 04021v1 Announce Type: cross Abstract: Proteolysis-targeting chimeras (PROTACs) can selectively degrade disease-causing proteins, yet predicting which targets are amenable to degradation remains a critical bottleneck: existing computational methods require the complete PROTAC molecular structure, information unavailable before synthesis.

By Bryan Cheng, Austin Jin

rag benchmarks

arXiv AI

Jun 4

No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

arXiv:2605. 30120v3 Announce Type: replace-cross Abstract: Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-level interactions.

By Lixuan Guo, Yifei Wang, Tiansheng Wen, Aosong Feng, Stefanie Jegelka, Chenyu You

rag efficiency benchmarks

arXiv AI

Jun 4

The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation

arXiv:2606. 04057v1 Announce Type: cross Abstract: Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions.

By Akanksha Narula, Mofasshara Binte Rafique, Laurent Bindschaedler

llms benchmarks

arXiv Machine Learning

Jun 4

VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

arXiv:2606. 04632v1 Announce Type: new Abstract: Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis.

By Teqi Hao, Yuxuan Fu, Xiaoyu Tan, Shaojie Shi, Bohao Lv, Yinghui Xu, Xihe Qiu

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 4

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

arXiv:2603. 23841v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact their objectivity.

By Rohan Khetan, Ashna Khetan

llms benchmarks safety

arXiv AI

Jun 4

Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning

arXiv:2605. 16301v2 Announce Type: replace-cross Abstract: Evaluating animal welfare reasoning in LLMs remains an open challenge despite rapid deployment in consumer and professional contexts where welfare considerations appear implicitly in everyday queries.

By Isabella Luong, Joyee Chen, Arturs Kanepajs, Jasmine Brazilek, Sankalpa Ghose, David Williams-King, Linh Le, Allen Lu

llms benchmarks safety

arXiv AI

Jun 4

MuCO: Generative Peptide Cyclization Empowered by Multi-stage Conformation Optimization

arXiv:2602. 11189v2 Announce Type: replace-cross Abstract: Modeling peptide cyclization is critical for the virtual screening of candidate peptides with desirable physical and pharmaceutical properties.

By Yitian Wang, Fanmeng Wang, Angxiao Yue, Wentao Guo, Yaning Cui, Hongteng Xu

benchmarks

arXiv AI

Jun 4

GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

arXiv:2603. 22121v2 Announce Type: replace-cross Abstract: Video Corpus Moment Retrieval (VCMR) aims to retrieve both the correct video and its temporal segment corresponding to a natural-language query, a task that is especially challenging for multi-verb queries where temporal action ordering is critical.

By Yunzhuo Sun, Xinyue Liu, Yanyang Li, Nanding Wu, Linlin Zong, Xianchao Zhang, Wenxin Liang

llms multimodal benchmarks

arXiv AI

Jun 4

Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space

arXiv:2604. 04944v2 Announce Type: replace-cross Abstract: Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs).

By Mohammad Reza Ghasemi Madani, Soyeon Caren Han, Shuo Yang, Jey Han Lau

llms benchmarks safety

arXiv AI

Jun 4

HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series

arXiv:2605. 11130v4 Announce Type: replace-cross Abstract: Critical events in multivariate time series, from turbine failures to cardiac arrhythmias, demand accurate prediction, yet labeled data is scarce because such events are rare and costly to annotate.

By Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni, Camilla Mazzoleni, Federico Martelli, Philipp Petersen

llms rag benchmarks

arXiv AI

Jun 4

Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problems

arXiv:2606. 04816v1 Announce Type: new Abstract: Large language models (LLMs) increasingly translate natural-language optimization problems into executable solver code.

By Xizi Luo, Changhong He, Dongdong Geng, Chenggong Shi, Yu Mei

llms benchmarks

arXiv AI

Jun 4

DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning

arXiv:2509. 10247v1 Announce Type: cross Abstract: This letter introduces DiffAero, a lightweight, GPU-accelerated, and fully differentiable simulation framework designed for efficient quadrotor control policy learning.

By Xinhong Zhang, Runqing Wang, Yunfan Ren, Jian Sun, Hao Fang, Jie Chen, Gang Wang

agents efficiency benchmarks

arXiv AI

Jun 4

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv:2606. 04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids.

By Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini

llms multimodal benchmarks

arXiv AI

Jun 4

SciDER: Scientific Data-centric End-to-end Researcher

arXiv:2603. 01421v3 Announce Type: replace Abstract: While large language models accelerate scientific discovery, existing agents face severe limitations in adaptability, domain generalization, and multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data.

By Ke Lin, Owais Aijaz, Yilin Lu, Yiyang Luo, Xuehang Guo, Preslav Nakov

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jun 4

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.

By Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan He, Zenan Li, Chi Jin, Venugopal V. Veeravalli, Aarti Gupta, Sanjeev Arora

llms benchmarks

arXiv AI

Jun 4

NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

arXiv:2606. 04806v1 Announce Type: cross Abstract: LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior.

By Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar

llms agents multimodal benchmarks safety

Get the daily briefing