Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv AI

Jun 4

Adaptive Minds: Empowering Agents with LoRA-as-Tools

arXiv:2510. 15416v2 Announce Type: replace Abstract: We investigate a framework in which LoRA adapters are treated as callable tools that a base language model can dynamically select and invoke.

By Pavan C Shekar, Aswanth Krishnan

llms agents fine-tuning benchmarks

arXiv AI

Jun 4

Knowledge Index of Noah's Ark

arXiv:2606. 05104v1 Announce Type: new Abstract: Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consensus; and unaudited ranking instability under bounded test budgets.

By Sheng Jin, Minghao Liu, Yunze Xiao, Zeqi Zhou, Heli Qi, Yifan Yao, Meishu Song, Kaijing Ma, Xuan Zhang, Sicong Jiang, Yizhe Li, Ningshan Ma, Jie Wei, Ziniu Li, Minglai Yang, Bangya Liu, Yiming Liang, Xiao Fang, Qingcheng Zeng, Jiarui Liu, Rui Yang, Shen Yan, Wenhao Huang, Jiaheng Liu, Zihan Wang, Weihao Xuan, Ge Zhang

llms benchmarks

arXiv AI

Jun 4

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv:2606. 04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids.

By Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini

llms multimodal benchmarks

arXiv AI

Jun 4

Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?

arXiv:2506. 10912v4 Announce Type: replace Abstract: Toxicity remains a leading cause of early-stage drug development failure.

By Fei Lin, Ziyang Gong, Cong Wang, Tengchao Zhang, Yonglin Tian, Yining Jiang, Ji Dai, Chao Guo, Xiaotong Yu, Xue Yang, Gen Luo, Fei-Yue Wang

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Training-Free Lexical-Dense Fusion for Conversational-Memory Retrieval

arXiv:2606. 04194v1 Announce Type: new Abstract: Retrieving the few past turns that answer a new query across long multi-session histories is the retrieval bottleneck behind long-term conversational memory (LoCoMo, LongMemEval).

By Christian Lysenst{\o}en

rag benchmarks safety

arXiv AI

Jun 4

Invariant Gradient Alignment for Robust Reasoning Distillation

arXiv:2606. 05025v1 Announce Type: cross Abstract: Large language models (LLMs) suffer from shortcut learning: they systematically fail on out-of-distribution (OOD) inputs whose semantic surface differs from training data, even when the logical structure is identical.

By Zehua Cheng, Wei Dai, Jiahao Sun

llms fine-tuning efficiency benchmarks safety

arXiv AI

Jun 4

M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

arXiv:2606. 05008v1 Announce Type: cross Abstract: As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability.

By Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong

multimodal benchmarks

arXiv AI

Jun 4

Self-Reflective APIs: Structure Beats Verbosity for AI Agent Recovery

arXiv:2606. 05037v1 Announce Type: cross Abstract: When an AI agent calls an API and hits a validation error, it needs more than what went wrong -- it needs what to do next.

By Arquimedes Canedo, Grama Chethan

llms agents benchmarks safety

arXiv Machine Learning

Jun 4

MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers

arXiv:2606. 04366v1 Announce Type: new Abstract: Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features.

By Yanshun Zhao, Xiaoyu Peng, Jiamin Jiang, Congcong Zhu, Jingrun Chen

llms nlp benchmarks safety

arXiv Machine Learning

Jun 4

Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning

arXiv:2606. 04492v1 Announce Type: new Abstract: Cooperative Multi-Agent Reinforcement Learning (MARL) frequently suffers from severe reward sparsity and exploration bottlenecks.

By Zicheng Zhao, Yu Lan, Chengzhengxu Li, Zhaohan Zhang, Xiaoming Liu

agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 4

VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

arXiv:2606. 04632v1 Announce Type: new Abstract: Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis.

By Teqi Hao, Yuxuan Fu, Xiaoyu Tan, Shaojie Shi, Bohao Lv, Yinghui Xu, Xihe Qiu

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 4

ALINC: Active Learning for Inductive Node Classification via Graph Sampling

arXiv:2606. 04647v1 Announce Type: new Abstract: Active learning (AL) for node classification typically focuses on selecting the most informative nodes for annotation within one or a few large graphs (e.

By Pascal Plettenberg, Denis Huseljic, Andr\'e Alcalde, Bernhard Sick, Josephine M. Thomas

benchmarks

arXiv AI

Jun 4

Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models

arXiv:2606. 04922v1 Announce Type: cross Abstract: Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging, where clinical data sensitivity favors frozen backbones and annotations are limited.

By Tran Dinh Tien, Zhiqiang Shen

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Rethinking Incompleteness: Formalizing Protocol Divergence and Train-Once Learning for Robust IMVC

arXiv:2606. 04857v1 Announce Type: new Abstract: Standard IMVC evaluation retrains separate models for different missing-data configurations.

By Haolu Liu, Xiyue Wang, Xuanting Xie, Liangjian Wen, Zhao Kang

llms benchmarks

arXiv Machine Learning

Jun 4

Variance-Gated Ensembles: An Epistemic-Aware Framework for Uncertainty Estimation

arXiv:2602. 08142v2 Announce Type: replace Abstract: Machine learning applications require fast and reliable per-sample uncertainty estimation.

By H. Martin Gillis, Isaac Xu, Thomas Trappenberg

benchmarks

arXiv Machine Learning

Jun 4

Efficient Reasoning on the Edge

arXiv:2603. 16867v2 Announce Type: replace Abstract: Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment.

By Yelysei Bondarenko, Thomas Hehn, Rob Hesselink, Romain Lepert, Fabio Valerio Massoli, Evgeny Mironov, Leyla Mirvakhabova, Tribhuvanesh Orekondy, Spyridon Stasis, Andrey Kuzmin, Anna Kuzina, Markus Nagel, Ankita Nayak, Corrado Rainone, Ork de Rooij, Paul N Whatmough, Arash Behboodi, Babak Ehteshami Bejnordi

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 4

REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

arXiv:2605. 20654v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process.

By Jiachen Ma, Jiawen Zhang, Xiangtian Li, Bo Zou, Chaochao Lu, Chao Yang

llms agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 4

RUBAS: Rubric-Based Reinforcement Learning for Agent Safety

arXiv:2606. 04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation.

By Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 4

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

arXiv:2606. 04613v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited.

By Alessandro Gambetti, Qiwei Han, Cl\'audia Soares, Hong Shen

llms multimodal benchmarks safety

arXiv AI

Jun 4

Scaling Novel Graph Generation via Lightweight Structure-Guided Autoregressive Models

arXiv:2606. 04287v1 Announce Type: cross Abstract: Generating realistic and diverse graphs is a key problem in machine learning, with applications in molecular discovery, circuit design, cybersecurity, and beyond.

By Alessio Barboni, Massimiliano Lupo Pasini, Bishal Lakha, Edoardo Serra

diffusion benchmarks

Get the daily briefing