Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv Machine Learning

Jun 4

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

arXiv:2603. 09803v2 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance.

By Tiehua Mei, Minxuan Lv, Leiyu Pan, Zhenpeng Su, Hongru Hou, Hengrui Chen, Ao Xu, Deqing Yang

llms reinforcement-learning benchmarks

arXiv AI

Jun 4

ANN Search: Recall What Matters

arXiv:2606. 04522v1 Announce Type: cross Abstract: Approximate nearest neighbor (ANN) search has become a core primitive in information retrieval and modern machine learning tasks, from classification to retrieval-augmented generation.

By Dimitris Dimitropoulos, Nikos Mamoulis

llms rag efficiency benchmarks

arXiv AI

Jun 4

Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learning

arXiv:2606. 04528v1 Announce Type: cross Abstract: Few-shot class-incremental learning (FSCIL) in synthetic aperture radar imagery presents unique challenges due to severe data scarcity and SAR-specific variability.

By Fan Zhang, Sijin Zheng, Fei Ma, Qiang Yin, Yongsheng Zhou, Fei Gao, Xian Sun

benchmarks

arXiv AI

Jun 4

Learning Long Range Spatio-Temporal Representations over Continuous Time Dynamic Graphs with State Space Models

arXiv:2606. 04672v1 Announce Type: cross Abstract: Continuous-time dynamic graphs (CTDGs) provide a richer framework to capture fine-grained temporal patterns in evolving relational data.

By Ayushman Raghuvanshi, Thummaluru Siddartha Readdy, Sundeep Prabhakar Chepuri, Mahesh Chandran

benchmarks

arXiv AI

Jun 4

ChessMimic: Per-Rating Transformer Models for Human Move, Clock, and Outcome Prediction in Online Blitz Chess

arXiv:2606. 04473v1 Announce Type: cross Abstract: We present ChessMimic, a system of three small encoder-only transformers - for move, thinking-time, and outcome prediction - conditioned on the position, recent move history, player rating, and clock state.

By Thomas Johnson

llms benchmarks safety

arXiv AI

Jun 4

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.

By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

agents benchmarks safety

arXiv AI

Jun 4

An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers

arXiv:2606. 04752v1 Announce Type: cross Abstract: Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step.

By Ossi Lehtinen

llms benchmarks

arXiv AI

Jun 4

Characterizing initial human-AI proof formalization workflows

arXiv:2606. 04273v1 Announce Type: new Abstract: For centuries, human mathematicians have written proofs to substantiate their mathematical arguments; yet, the ability to automatically verify the validity of proofs has long been a challenge.

By Katherine M. Collins, Simon Frieder, Jonas Bayer, Jacob Loader, Jeck Lim, Peiyang Song, Fabian Zaiser, Lexin Zhou, Shanda Li, Sam Looi, Joshua B. Tenenbaum, Umang Bhatt, Adrian Weller, Jose Hernandez-Orallo, Cameron E. Freer, Valerie Chen, Ilia Sucholutsky

benchmarks

arXiv AI

Jun 4

Self-Evolving Deep Research via Joint Generation and Evaluation

arXiv:2606. 04507v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability.

By Han Zhu, Chengkun Cai, Yuanfeng Song, Xing Chen, Sirui Han, Yike Guo

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 4

Learning Randomized Reductions

arXiv:2412. 18134v5 Announce Type: replace Abstract: Randomized self-reductions (RSRs) express $f(x)$ using $f$ evaluated at random correlated points, enabling self-correcting programs, instance-hiding protocols, and applications in complexity theory and cryptography.

By Ferhat Erata, Orr Paradise, Thanos Typaldos, Timos Antonopoulos, ThanhVu Nguyen, Shafi Goldwasser, Ruzica Piskac

llms agents benchmarks

arXiv AI

Jun 4

MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models

arXiv:2606. 04027v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) generate text by iteratively denoising partially masked sequences under bidirectional context, exposing a safety surface distinct from autoregressive LLMs.

By Yingzi Ma, Zhengyue Zhao, Xiaogeng Liu, Minhui Xue, Yue Zhao, Chaowei Xiao

llms diffusion reinforcement-learning benchmarks safety

arXiv AI

Jun 4

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

arXiv:2606. 04442v1 Announce Type: cross Abstract: AI systems increasingly need to combine two demanding capabilities: navigating multi-session conversation history and performing deep reading comprehension within long documents.

By Qiyang Xie, Jialun Wu, Xinjie He, Su Liu, Shuai Xiao, Zhiyuan Lin, Weikai Zhou

llms rag benchmarks

arXiv AI

Jun 4

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.

By Devleena Das, Rajeev Patwari, Elliott Delaye, Ashish Sirasao

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

HEPA: A Self-Supervised Horizon-Conditioned Event Predictive Architecture for Time Series

arXiv:2605. 11130v4 Announce Type: replace-cross Abstract: Critical events in multivariate time series, from turbine failures to cardiac arrhythmias, demand accurate prediction, yet labeled data is scarce because such events are rare and costly to annotate.

By Jonas Petersen, Gian-Alessandro Lombardi, Riccardo Maggioni, Camilla Mazzoleni, Federico Martelli, Philipp Petersen

llms rag benchmarks

arXiv AI

Jun 4

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

arXiv:2606. 04280v1 Announce Type: cross Abstract: Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood.

By Justinas Zaliaduonis, Patrick Putzky, Till Richter, Sergios Gatidis

diffusion benchmarks safety

arXiv AI

Jun 4

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

arXiv:2506. 06006v3 Announce Type: replace-cross Abstract: Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.

By Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 4

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv:2606. 04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids.

By Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini

llms multimodal benchmarks

arXiv AI

Jun 4

Toward Pre-Deployment Assurance for Enterprise AI Agents: Ontology-Grounded Simulation and Trust Certification

arXiv:2606. 04037v1 Announce Type: new Abstract: Pre-deployment verification of enterprise artificial intelligence (AI) agents remains a critical gap between large language model (LLM) capability benchmarking and production deployment.

By Thanh Luong Tuan, Abhijit Sanyal

llms rag agents benchmarks safety

arXiv Machine Learning

Jun 4

Post-Training Corrections for Improved Time-Series Forecasting

arXiv:2505. 15354v3 Announce Type: replace Abstract: Time-series forecasting is a critical task in various business domains, but it remains inherently challenging.

By Hamza Cherkaoui, Malik Tiomoko, Giuseppe Paolo, Zhang Yili, Yu Meng, Zhang Keli, Hafiz Tiomoko Ali

benchmarks

Get the daily briefing