Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv AI

Jun 19

Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

arXiv:2606. 20177v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in various Remote Sensing (RS) tasks.

By Haochen Han, Jue Wang, Alex Jinpeng Wang, Fangming Liu

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Where Does Social Reasoning Come From? Capability Provenance in Language Models

arXiv:2606. 19625v1 Announce Type: cross Abstract: We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B.

By Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl

llms benchmarks

arXiv AI

Jun 19

Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

arXiv:2606. 20363v1 Announce Type: new Abstract: Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies.

By Yuexing Hao, Xiaomin Li

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

arXiv:2606. 20537v1 Announce Type: new Abstract: Mainstream LLM serving systems reuse prefix work mainly through paged or radix key-value (KV) caches.

By Liang Su

llms agents robotics efficiency benchmarks data-engineering

arXiv AI

Jun 19

Process-Verified Reinforcement Learning for Theorem Proving via Lean

arXiv:2606. 20068v1 Announce Type: new Abstract: While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback.

By Minsu Kim, Se-Young Yun

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

EFIQA: Explainable Fundus Image Quality Assessment via Anatomical Priors

arXiv:2606. 20108v1 Announce Type: cross Abstract: Image quality control is vital for a wide range of downstream applications.

By Pengwei Wang, Jos\'e Morano, Qian Wan, Hrvoje Bogunovi\'c

benchmarks

arXiv AI

Jun 19

VOiLA: Vectorized Online Planning with Learned Diffusion Model for POMDP Agents

arXiv:2606. 19729v1 Announce Type: cross Abstract: Planning under uncertainty is an essential capability for autonomous robots.

By Marcus Hoerger, Rishikesh Joshi, Rahul Shome, Ian Manchester, Hanna Kurniawati

agents diffusion reinforcement-learning robotics efficiency benchmarks

arXiv AI

Jun 19

Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.

By Xuanzhi Feng, Zhengyang Li, Zeyu Liu, Haoxi Li, Yuming Jiang, Bing Guo, Jingcai Guo, Jie Zhang, Song Guo

llms reinforcement-learning benchmarks

arXiv AI

Jun 19

Uncertainty Decomposition for Clarification Seeking in LLM Agents

arXiv:2606. 19559v1 Announce Type: new Abstract: Recent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for interactive large language model (LLM) agents and call for underspecification-aware, decomposed, and communicable uncertainty representations that can unlock new agent capabilities such as proactive clarification seeking and shared mental-model building.

By Gregory Matsnev

llms agents benchmarks

arXiv AI

Jun 19

CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation

arXiv:2605. 10873v2 Announce Type: replace-cross Abstract: Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existing evaluations are fragmented across datasets, modalities, and metrics.

By Anna C. Doris, Jacob Thomas Sony, Ghadi Nehme, Era Syla, Amin Heyrani Nobari, Faez Ahmed

multimodal benchmarks

arXiv Machine Learning

Jun 19

UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

arXiv:2606. 20559v1 Announce Type: cross Abstract: Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action.

By Wenhao Chi, Arkaprava Sinha, Dominick Reilly, Hieu Le, Srijan Das

computer-vision efficiency benchmarks

arXiv AI

Jun 19

Hybrid ANN-SNN Pipeline with Local Plasticity

arXiv:2606. 20151v1 Announce Type: cross Abstract: This work proposes a hybrid ANN-SNN pipeline that effectively leverages the rich embeddings of pretrained artificial neural networks (ANNs) to enable high-performance spiking neural networks (SNNs).

By Denis Larionov, Khairutin Shtanchaev, Mikhail Kiselev, Mikhail Korovin, Ivan Tugoy

rag benchmarks

arXiv Machine Learning

Jun 19

A Unified Perspective on the Dynamics of Deep Transformers

arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.

By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e

llms benchmarks

arXiv Machine Learning

Jun 19

PaAno+: Multiscale Encoding and Cross-Variable Attention for Time Series Anomaly Detection

arXiv:2606. 20055v1 Announce Type: new Abstract: Time-series anomaly detection has significant practical value for industrial and medical monitoring, as well as other critical domains.

By Youji Zhu, Hongbing Wang, Wenchao Liu, Xiaodong Liu, Xiangguang Xiong

llms rag benchmarks

arXiv AI

Jun 19

The Autonomy Tax: Defense Training Breaks LLM Agents

arXiv:2603. 19423v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks.

By Shawn Li, Yue Zhao

llms agents benchmarks safety

arXiv Machine Learning

Jun 19

MolGraphBench: A Benchmark of GNN Architectures for Molecular Regression Tasks

arXiv:2602. 20573v3 Announce Type: replace Abstract: Molecules are often represented as SMILES strings, which can be readily converted to hand-crafted descriptors or fingerprints (FP) for molecular property prediction.

By Rajan, Ishaan Gupta

fine-tuning benchmarks

arXiv AI

Jun 19

MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer

arXiv:2606. 20094v1 Announce Type: cross Abstract: Makeup transfer models enable fun augmented reality (AR) experiences as well as virtual try-on (VTO) for online makeup shopping.

By Nefeli Andreou, Angel Mart\'inez-Gonz\'alez, Sabine Sternig, Matthieu Guillaumin, Epameinondas Antonakos, Michael Opitz

diffusion benchmarks

arXiv AI

Jun 19

CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models

arXiv:2606. 19788v1 Announce Type: new Abstract: We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models.

By Yuxu Zhou, Ond\v{r}ej Ku\v{z}elka, Yuyi Wang, Yuanhong Wang, Yi Chang

llms benchmarks

arXiv AI

Jun 19

Reinforcement-aware Knowledge Distillation for LLM Reasoning

arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.

By Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 19

Structure-Oriented Randomized Neural Networks for Poisson-Nernst-Planck and Poisson-Nernst-Planck-Navier-Stokes Systems

arXiv:2606. 19912v1 Announce Type: cross Abstract: We develop a structure-oriented randomized neural network framework, termed SO-RaNN, for the Poisson-Nernst-Planck (PNP) system and the Poisson-Nernst-Planck-Navier-Stokes (PNP-NS) system.

By Yunlong Li, Fei Wang

benchmarks

Get the daily briefing