Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,859 stories · RSS feed

arXiv AI

Jun 15

PRISM: Perception Reasoning Interleaved for Sequential Decision Making

arXiv:2605. 05407v2 Announce Type: replace Abstract: Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge.

By Mohamed Salim Aissi, Clemence Grislain, Clement Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, Nicolas Thome

llms agents robotics multimodal benchmarks

arXiv AI

Jun 15

COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

arXiv:2512. 02318v4 Announce Type: replace-cross Abstract: This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA.

By Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Mingkui Wei, Junjie Xiong

llms multimodal benchmarks

arXiv AI

Jun 15

Selective Agentic Recovery for UAV Autonomy with a Persistent Mission Runtime

arXiv:2606. 14219v1 Announce Type: cross Abstract: Agentic AI can support unmanned aerial vehicle (UAV) autonomy by providing high-level recovery reasoning when local waypoint- or setpoint-based execution encounters blocked passages, repeated no-progress behavior, or mission-level ambiguity.

By Taewoo Park, Kyeonghyun Yoo, Seunghyun Yoo, Hwangnam Kim

agents benchmarks safety

arXiv Machine Learning

Jun 15

Utility-Constrained Policy Optimization

arXiv:2606. 14029v1 Announce Type: new Abstract: Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints.

By Mehrdad Moghimi, Bernardo Avila Pires

agents reinforcement-learning benchmarks safety

arXiv AI

Jun 15

CisTransCell: Single-Cell Perturbation Prediction via Gene Function, Regulatory Control, and Cellular Context

arXiv:2606. 13713v1 Announce Type: cross Abstract: Predicting cellular transcriptional responses to genetic perturbations is a central problem in single-cell biology, especially in the zero-shot setting where the perturbed gene or gene combination is unseen during training.

By Wei Zhang, Xun Jiang, Yuesi Xi, Ming Tang

multimodal benchmarks

arXiv Machine Learning

Jun 15

Recipe-Controlled Decoder Audit for Structural Knowledge-Graph Completion

arXiv:2606. 14492v1 Announce Type: new Abstract: We present a recipe-controlled decoder audit (RCDA) for structural transductive knowledge-graph completion (KGC).

By Xihang Shan, Ye Luo

benchmarks

arXiv AI

Jun 15

3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding

arXiv:2603. 04976v2 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Rewards ( RLVR ) has emerged as a transformative paradigm for enhancing the reasoning capabilities of Large Language Models ( LLMs), yet its potential in 3D scene understanding remains under-explored.

By Xiongkun Linghu, Jiangyong Huang, Baoxiong Jia, Siyuan Huang

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jun 15

Numbers Already Carry Their Own Embeddings

arXiv:2606. 14108v1 Announce Type: cross Abstract: We introduce Adelic operation-preserved embeddings (AOE), a training-free representation that captures both a number's real value and its modular (p-adic) signatures.

By Suhyun Bae, Donghun Lee

rag benchmarks

arXiv Machine Learning

Jun 15

Curvature-Informed Potential Energy Surface for Protein-Ligand Binding Affinity Prediction

arXiv:2606. 14217v1 Announce Type: new Abstract: Accurate prediction of protein-ligand binding affinity is essential for structure-based drug discovery.

By Peng-Fei Sun, Chuan-Xian Ren, Hong Yan

benchmarks safety

arXiv AI

Jun 15

Refusal Beyond a Single Direction: A Preliminary Comparison of Diff-in-Means and INLP

arXiv:2606. 13720v1 Announce Type: new Abstract: Arditi et al.

By Elisabetta Rocchetti, Alfio Ferrara

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 15

Decompose Sparsely Where You Should, Absorb Densely Where You Should No

arXiv:2606. 14040v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are typically trained to reconstruct the \textbf{entire} residual stream through a sparse dictionary, implicitly assuming that all activation content is amenable to sparse, monosemantic decomposition.

By Ruixuan Deng, Zehao Jin, Zekun Wang, Zihan Dong

efficiency benchmarks safety

arXiv AI

Jun 15

AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

arXiv:2606. 14295v1 Announce Type: cross Abstract: Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation.

By Fengyu Liu, Jiarun Dai, Yihe Fan, Wuyuao Mai, Ziao Li, Bofei Chen, Jie Zhang, Zheng Lou, Bocheng Xiang, Qiyi Zhang, Xudong Pan, Geng Hong, Yuan Zhang, Min Yang

llms agents benchmarks

arXiv AI

Jun 15

Learning optimal policies from event logs through reinforcement learning: a comparison of deep and MDP-based approaches

arXiv:2303. 09209v2 Announce Type: replace Abstract: Prescriptive Process Monitoring is an emerging area within Process Mining that focuses on recommending actions to optimize business outcomes.

By Stefano Branchi, Andrei Buliga, Chiara Di Francescomarino, Chiara Ghidini, Riccardo Graziosi, Francesca Meneghello, Massimiliano Ronzani

reinforcement-learning benchmarks

arXiv AI

Jun 15

SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

arXiv:2606. 14574v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as planners for autonomous agents in household environments.

By Xiaoxin Lu, Ranran Haoran Zhang, Rui Zhang

llms agents benchmarks

arXiv AI

Jun 15

Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

arXiv:2602. 03120v2 Announce Type: replace-cross Abstract: Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune.

By Yinggan Xu, Kajetan Schweighofer, Risto Miikkulainen, Xin Qiu

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 15

From Sorting Algorithms to Scalable Kernels: Bayesian Optimization in High-Dimensional Permutation Spaces

arXiv:2507. 13263v4 Announce Type: replace-cross Abstract: Bayesian Optimization (BO) is a powerful tool for black-box optimization, but its application to high-dimensional permutation spaces is severely limited by the challenge of defining scalable representations.

By Zikai Xie, Linjiang Chen

benchmarks

arXiv AI

Jun 15

DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining

arXiv:2606. 14283v1 Announce Type: cross Abstract: Deep learning has driven many recent advances in process analytics, especially for predictive and prescriptive monitoring.

By Johannes De Smedt, Jari Peeperkorn, Artem Polyvyanyy, Jochen De Weerdt

llms benchmarks

arXiv AI

Jun 15

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

arXiv:2606. 14120v1 Announce Type: cross Abstract: Auditory attention decoding (AAD) aims to infer the attended speaker from neural responses in multi-speaker acoustic environments and is a key problem for neuro-steered hearing systems.

By Ziwei Wang, Xingyi He, Tianwang Jia, Hongbin Wang, Dongrui Wu

llms benchmarks safety

arXiv AI

Jun 15

Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Search

arXiv:2601. 04646v4 Announce Type: replace-cross Abstract: Large-scale multi-tenant retrieval systems generate extensive query logs but lack curated relevance labels for effective domain adaptation, resulting in substantial underutilized "dark data.

By Prateek Jain, Shabari S Nair, Ritesh Goru, Prakhar Agarwal, Ajay Yadav, Yoga Sri Varshan Varadharajan, Constantine Caramanis

llms fine-tuning benchmarks

arXiv AI

Jun 15

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

arXiv:2504. 20734v5 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries.

By Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

rag fine-tuning benchmarks

Get the daily briefing