Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv Machine Learning

Jun 3

Forecasting Conceptual Diffusion in Science: The Case of Quantum Computing

arXiv:2606. 03919v1 Announce Type: cross Abstract: Understanding and anticipating scientific change requires models that distinguish between endogenous consolidation and exogenous diffusion of scientific concepts.

By Thomas Maillart, Thibaut Chataing, David Dosu, Paul Bagourd, Julian Jang-Jaccard, Alain Mermoud

diffusion robotics benchmarks

arXiv Machine Learning

Jun 3

WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

arXiv:2603. 04956v2 Announce Type: replace Abstract: This paper considers the problem of converting a given dense linear layer to low precision.

By Egor Lifar, Semyon Savkin, Or Ordentlich, Yury Polyanskiy

llms efficiency benchmarks

arXiv Machine Learning

Jun 3

Visual Instruction Tuning Aligns Modalities through Abstraction

arXiv:2606. 03871v1 Announce Type: cross Abstract: Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text.

By Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga

llms rag fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 3

QUIVER: Quantum-Informed Views for Enhanced Representations in Large ML Models

arXiv:2606. 02785v1 Announce Type: new Abstract: Large machine learning models benefit substantially from multimodal inputs that provide a complementary view of the same example.

By Aritra Bal, Michael Binder, Markus Klute, Benedikt Maier, Michael Spannowsky

multimodal benchmarks

arXiv Machine Learning

Jun 3

A Nonmonotone Gradient-Based Algorithm for Symmetric Nonnegative Matrix Factorization and Graph Clustering

arXiv:2606. 02887v1 Announce Type: new Abstract: Symmetric nonnegative matrix factorization (Symmetric NMF) approximates a matrix as $WW^T$ with nonnegative rectangular factor $W$.

By Ryan Swart, Johannes Brust

benchmarks

arXiv AI

Jun 3

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

arXiv:2606. 03988v1 Announce Type: new Abstract: Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable.

By Mahtab Bigverdi, Lindsey Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dangjoo Kim, Zelun Luo, Linda Shapiro, Ranjay Krishna

llms multimodal benchmarks

arXiv Machine Learning

Jun 3

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

arXiv:2606. 03184v1 Announce Type: cross Abstract: Financial forecasting is difficult due to low signal-to-noise ratios, latent factors, heavy tails, regime shifts, and jumps.

By Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni, Yize Liu, Haonan Chen, Ke-Wei Huang

llms multimodal benchmarks safety

arXiv AI

Jun 3

LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks

arXiv:2606. 03303v1 Announce Type: new Abstract: Large Language Models (LLMs) exhibit strong informal mathematical reasoning but struggle to generate mechanically verifiable proofs in formal languages like Lean.

By Po-Nien Kung, Linfeng Song, Dawsen Hwang, Jinsung Yoon, Chun-Liang Li, Simone Severini, Mirek Ol\v{s}\'ak, Edward Lockhart, Quoc V Le, Burak Gokturk, Thang Luong, Tomas Pfister, Nanyun Peng

llms agents benchmarks

arXiv Machine Learning

Jun 3

Neutrino Fingerprints: Image-Based Encodings of IceCube Events for CNN Direction Reconstruction

arXiv:2606. 02788v1 Announce Type: cross Abstract: Reconstructing the direction of incoming neutrinos in the IceCube Neutrino Observatory is an important problem in astrophysics.

By Floriano Tori, Brecht Verbeken, Vincent Ginis

benchmarks

arXiv Machine Learning

Jun 3

ScoreStop: Gradient-based early stopping using functional score tests

arXiv:2606. 02740v1 Announce Type: cross Abstract: Gradient boosted decision trees require a stopping rule to avoid overfitting.

By Oliver J. Hines, Christian L. Hines

benchmarks

arXiv AI

Jun 3

Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement

arXiv:2412. 01282v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks.

By Qianhan Feng, Wenshuo Li, Tong Lin, Xinghao Chen

llms rag efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 3

High-Dimensional Latents Should Be Diagnosed Through Phase Structure

arXiv:2606. 02600v1 Announce Type: cross Abstract: We study autoencoder and variational-autoencoder latent spaces through the lens of spin-glass theory.

By Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, Olivier Salvado

benchmarks

arXiv AI

Jun 3

AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

arXiv:2606. 02461v2 Announce Type: replace Abstract: Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes.

By Yiheng Shu, Bernal Jim\'enez Guti\'errez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su

agents benchmarks

arXiv AI

Jun 3

Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge

arXiv:2512. 03019v2 Announce Type: replace-cross Abstract: Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed.

By Hamid Dadkhahi, Firas Trabelsi, Parker Riley, Juraj Juraska, Mehdi Mirzazadeh

llms benchmarks

arXiv Machine Learning

Jun 3

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

arXiv:2606. 03980v1 Announce Type: new Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines.

By Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 3

AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse

arXiv:2605. 03644v2 Announce Type: replace Abstract: Many-Shot In-Context Learning (ICL) has emerged as a promising paradigm, leveraging extensive examples to unlock the reasoning potential of Large Language Models (LLMs).

By Jie Ou, Jinyu Guo, Shiyao Guo, Yuang Li, Ruiqi Wu, Zhaokun Wang, Wenyi Li, Wenhong Tian

llms efficiency benchmarks

arXiv AI

Jun 3

Towards a Science of AI Agent Reliability

arXiv:2602. 16666v3 Announce Type: replace Abstract: AI agents are increasingly deployed to execute important tasks.

By Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan

agents benchmarks safety

arXiv AI

Jun 3

FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaboration

arXiv:2512. 11213v2 Announce Type: replace Abstract: Scaling test-time computation has been shown to significantly improve large language model (LLM) performance without additional training.

By Dongwon Jung, Peng Shi, Muhao Chen, Yi Zhang

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 3

ProtocolBench: Which LLM MultiAgent Protocol to Choose?

arXiv:2510. 17149v3 Announce Type: replace Abstract: As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability.

By Hongyi Du, Jiaqi Su, Jisen Li, Lijie Ding, Yingxuan Yang, Peixuan Han, Xiangru Tang, Kunlun Zhu, Jiaxuan You

llms agents benchmarks

arXiv AI

Jun 3

Automatic Layer Selection for Hallucination Detection

arXiv:2605. 26366v3 Announce Type: replace Abstract: Recent studies on hallucination detection have shown that hallucination-related signals are more strongly encoded in intermediate layers than in the final layer of large language models (LLMs).

By Xinpeng Wang, William X. Cao, Andrew Gordon Wilson, Zhe Zeng

llms nlp benchmarks safety

Get the daily briefing