Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv AI

Jun 4

ClustRecNet: A Novel End-to-End Deep Learning Framework for Clustering Algorithm Recommendation

arXiv:2509. 25289v4 Announce Type: replace-cross Abstract: Identifying an effective clustering algorithm for a given dataset remains a fundamental unsupervised learning issue.

By Mohammadreza Bakhtyari, Bogdan Mazoure, Renato Cordeiro de Amorim, Guillaume Rabusseau, Vladimir Makarenkov

benchmarks

arXiv AI

Jun 4

Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

arXiv:2606. 04970v1 Announce Type: cross Abstract: We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding \textit{when} to interrupt, and \textit{how} to coach.

By Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Th\'eo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Pinar Donmez, Babak Damavandi, Anuj Kumar, Seungwhan Moon

llms multimodal benchmarks

arXiv AI

Jun 4

No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

arXiv:2605. 30120v3 Announce Type: replace-cross Abstract: Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-level interactions.

By Lixuan Guo, Yifei Wang, Tiansheng Wen, Aosong Feng, Stefanie Jegelka, Chenyu You

rag efficiency benchmarks

arXiv AI

Jun 4

Adaptive Minds: Empowering Agents with LoRA-as-Tools

arXiv:2510. 15416v2 Announce Type: replace Abstract: We investigate a framework in which LoRA adapters are treated as callable tools that a base language model can dynamically select and invoke.

By Pavan C Shekar, Aswanth Krishnan

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jun 4

Rethinking Incompleteness: Formalizing Protocol Divergence and Train-Once Learning for Robust IMVC

arXiv:2606. 04857v1 Announce Type: new Abstract: Standard IMVC evaluation retrains separate models for different missing-data configurations.

By Haolu Liu, Xiyue Wang, Xuanting Xie, Liangjian Wen, Zhao Kang

llms benchmarks

arXiv Machine Learning

Jun 4

ALINC: Active Learning for Inductive Node Classification via Graph Sampling

arXiv:2606. 04647v1 Announce Type: new Abstract: Active learning (AL) for node classification typically focuses on selecting the most informative nodes for annotation within one or a few large graphs (e.

By Pascal Plettenberg, Denis Huseljic, Andr\'e Alcalde, Bernhard Sick, Josephine M. Thomas

benchmarks

arXiv AI

Jun 4

SciDER: Scientific Data-centric End-to-end Researcher

arXiv:2603. 01421v3 Announce Type: replace Abstract: While large language models accelerate scientific discovery, existing agents face severe limitations in adaptability, domain generalization, and multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data.

By Ke Lin, Owais Aijaz, Yilin Lu, Yiyang Luo, Xuehang Guo, Preslav Nakov

llms agents fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 4

VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

arXiv:2606. 04632v1 Announce Type: new Abstract: Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis.

By Teqi Hao, Yuxuan Fu, Xiaoyu Tan, Shaojie Shi, Bohao Lv, Yinghui Xu, Xihe Qiu

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 4

REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

arXiv:2605. 20654v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process.

By Jiachen Ma, Jiawen Zhang, Xiangtian Li, Bo Zou, Chaochao Lu, Chao Yang

llms agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 4

Do LLMs Hold Their Values? MANTA: A Multi-Turn Adversarial Benchmark for Animal Welfare Reasoning

arXiv:2605. 16301v2 Announce Type: replace-cross Abstract: Evaluating animal welfare reasoning in LLMs remains an open challenge despite rapid deployment in consumer and professional contexts where welfare considerations appear implicitly in everyday queries.

By Isabella Luong, Joyee Chen, Arturs Kanepajs, Jasmine Brazilek, Sankalpa Ghose, David Williams-King, Linh Le, Allen Lu

llms benchmarks safety

arXiv AI

Jun 4

Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time

arXiv:2504. 12329v2 Announce Type: replace-cross Abstract: Recent advances leverage post-training to enhance model reasoning performance, which typically requires costly training pipelines and still suffers from inefficient, overly lengthy outputs.

By Wang Yang, Xiang Yue, Vipin Chaudhary, Xiaotian Han

benchmarks

arXiv AI

Jun 4

From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

arXiv:2512. 05277v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 4

On the Expressive Power of Permutation-Equivariant Weight-Space Networks

arXiv:2602. 01083v2 Announce Type: replace Abstract: Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks.

By Adir Dayan, Yam Eitan, Haggai Maron

benchmarks

arXiv AI

Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento

llms benchmarks

arXiv AI

Jun 4

BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation format

arXiv:2509. 02655v3 Announce Type: replace-cross Abstract: Many AI alignment discussions of "runaway optimisation" focus on RL agents: unbounded utility maximisers that over-optimise a proxy objective (e.

By Roland Pihlakas (for the Three Laws collaboration), Sruthi Susan Kuriakose (for the Three Laws collaboration)

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 4

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

arXiv:2601. 09719v3 Announce Type: replace-cross Abstract: Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning.

By Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, Kyungwoo Song

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 4

MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models

arXiv:2606. 04627v1 Announce Type: new Abstract: Mobile agents are increasingly expected to operate everyday applications from screenshots and language goals, where reliable control requires reasoning over screen affordances, multi-step navigation, and future state changes.

By Zhichao Yang, Yuanze Hu, Haojie Hao, Longkun Hao, Dongshuo Huang, Hongyu Lin, Gen Li, Lanqing Hong, Yihang Lou, Yan Bai

agents diffusion fine-tuning benchmarks

arXiv Machine Learning

Jun 4

The Perception-Physics Paradox: Probing Scientific Alignment with TC-Bench

arXiv:2605. 24782v2 Announce Type: replace Abstract: While Vision Foundation Models (VFMs) excel at predictive tasks on satellite imagery, their performance can arise from visual correlations rather than underlying structural invariants, making even perception-based out-of-distribution accuracy a poor proxy for scientific utility.

By Dingling Yao, Andrea Polesello, Adeel Pervez, Caroline Muller, Francesco Locatello

benchmarks safety

arXiv AI

Jun 4

LaVIDE: Language-Prompted Satellite Change Detection via Map-Image Alignment

arXiv:2411. 19758v2 Announce Type: replace-cross Abstract: Remote sensing change detection based on a map reference and an up-to-date image boosts timely observation of the Earth's surface when earlier images are lacking for comparison.

By Shuguo Jiang, Fang Xu, Chuandong Liu, Hong Tan, Shengyang Li, Lei Yu, Wen Yang, Sen Jia, Gui-Song Xia

rag computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Identifying and Correcting Label Noise for Robust GNNs via Influence Contradiction

arXiv:2601. 17469v2 Announce Type: replace Abstract: Graph Neural Networks (GNNs) have shown remarkable capabilities in learning from graph-structured data with various applications such as social analysis and bioinformatics.

By Wei Ju, Wei Zhang, Siyu Yi, Zhengyang Mao, Yifan Wang, Jingyang Yuan, Zhiping Xiao, Ziyue Qiao, Ming Zhang

diffusion benchmarks

Get the daily briefing