Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

AdaSTORM: Scaling LLM Reasoning on Dynamic Graphs via Adaptive Spatio-Temporal Multi-Agent Collaboration

arXiv:2606. 16328v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate remarkable potential in dynamic graph reasoning, but suffer from a scaling bottleneck: current models can only handle graphs with tens of nodes, constrained by exponential reasoning overhead and finite context windows.

By Bing Hao, Ruijie Wang, Haodong Qian, Yunlong Chu, Yuhang Liu, Yumeng Lin, Minglai Shao, Jianxin Li

llms agents benchmarks

arXiv AI

Jun 16

Training and Evaluating Diffusion Policies with Long Context Lengths

arXiv:2606. 16447v1 Announce Type: cross Abstract: Imitation learning has enabled highly-dexterous robotic manipulation from RGB observations.

By Abhinav Agarwal, Adam Wei, Taylan Kargin, Michael Zeng, Cole Becker, Arif Kerem Dayi, Pablo Parrilo, Asuman Ozdaglar, Russ Tedrake

diffusion robotics benchmarks

arXiv AI

Jun 16

RL-Index: Reinforcement Learning for Retrieval Index Reasoning

arXiv:2606. 16316v1 Announce Type: cross Abstract: Retrieving external knowledge is essential for solving real-world tasks, yet it remains challenging when the relationship between a query and its relevant knowledge involves implicit and complex reasoning beyond surface-level semantic or lexical matching (e.

By Yongjia Lei, Nedim Lipka, Zhisheng Qi, Utkarsh Sahu, Koustava Goswami, Franck Dernoncourt, Ryan A. Rossi, Yu Wang

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 16

Sharp analysis of linear ensemble sampling

arXiv:2602. 08026v2 Announce Type: replace Abstract: We analyse linear ensemble sampling (ES) with standard Gaussian perturbations in stochastic linear bandits.

By David Janz, Arya Akhavan, Csaba Szepesv\'ari

benchmarks

arXiv AI

Jun 16

G-Loss: Graph-Guided Fine-Tuning of Language Models

arXiv:2604. 25853v3 Announce Type: replace-cross Abstract: Traditional loss functions, including cross-entropy, contrastive, triplet, and su pervised contrastive losses, used for fine-tuning pre-trained language models such as BERT, operate only within local neighborhoods and fail to account for the global semantic structure.

By Aditya Sharma, Vinti Agarwal, Rajesh Kumar

llms rag nlp fine-tuning benchmarks

arXiv AI

Jun 16

Discovering Symmetry Groups with Flow Matching

arXiv:2512. 20043v3 Announce Type: replace Abstract: Symmetry is fundamental to understanding physical systems and can improve performance and sample efficiency in machine learning.

By Yuxuan Chen, Jung Yeon Park, Floor Eijkelboom, Jianke Yang, Jan-Willem van de Meent, Lawson L. S. Wong, Robin Walters

diffusion benchmarks

arXiv AI

Jun 16

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

arXiv:2606. 15258v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources.

By Jierui Zhang, Siyuan Tan, Xinhang Li, Longzhuangzhi Lin, Dailin Li, Chengfeng Gu, Xinping Li, Yaxian Hao, Shengjia Liang, Yuxiang Ren, Wenhao Liu

llms benchmarks

arXiv AI

Jun 16

Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw

arXiv:2605. 11047v2 Announce Type: replace-cross Abstract: Agentic language-model systems increasingly rely on mutable execution contexts, including files, memory, tools, skills, and auxiliary artifacts, creating security risks beyond explicit user prompts.

By Hongwei Yao, Yiming Liu, Yiling He, Bingrun Yang

agents robotics benchmarks safety

arXiv Machine Learning

Jun 16

HawkesNest: A Multi-Axis Synthetic Benchmark for Spatiotemporal Pattern Complexity

arXiv:2606. 16863v1 Announce Type: new Abstract: Evaluation of spatiotemporal point process (STPP) models relies heavily on opaque real-world datasets, where latent generative structure is unknown and model failures are difficult to attribute.

By Yahya Aalaila, Sumantrak Mukherjee, Gerrit Gro{\ss}mann, Sebastian Vollmer

benchmarks

arXiv AI

Jun 16

Scalable Circuit Learning for Interpreting Large Language Models

arXiv:2606. 16939v1 Announce Type: cross Abstract: A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior.

By Naiyu Yin, Dennis Wei, Tian Gao, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, Yue Yu

llms benchmarks safety

arXiv AI

Jun 16

LLM Jaggedness Unlocks Scientific Creativity

arXiv:2605. 10574v3 Announce Type: replace Abstract: As artificial intelligence advances, models are not improving uniformly.

By Shray Mathur, J. Anibal Boscoboinik, Esther H. R. Tsai, Kevin G. Yager

llms benchmarks

arXiv Machine Learning

Jun 16

A Mechanistic Understanding of Pronoun Fidelity in LLMs

arXiv:2606. 16407v1 Announce Type: cross Abstract: Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns.

By Katharina Trinley, Jesujoba O. Alabi, Dietrich Klakow, Vagrant Gautam

llms benchmarks safety

arXiv Machine Learning

Jun 16

Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacy

arXiv:2407. 04884v4 Announce Type: replace Abstract: The hidden state threat model of differential privacy (DP) assumes that the adversary has access only to the final trained machine learning (ML) model, without seeing intermediate states during training.

By Rob Romijnders, Antti Koskela

benchmarks safety

arXiv Machine Learning

Jun 16

How Should World Models Be Evaluated? A Decision-Making-Centric Position

arXiv:2606. 15032v1 Announce Type: new Abstract: World models have rapidly become one of the central abstractions in modern AI.

By Yang Yu, Shiyuan Zhang, Yifei Sheng, Haoxiang Ren, Haoxin Lin

robotics benchmarks

arXiv Machine Learning

Jun 16

MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

arXiv:2606. 16748v1 Announce Type: new Abstract: Current benchmarks for computer-use agents evaluate models in impersonal environments.

By Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

llms agents benchmarks

arXiv Machine Learning

Jun 16

A nonparametric two-sample test using a parametric integral probability metric

arXiv:2606. 16941v1 Announce Type: cross Abstract: Detecting distributional differences between two independent samples is a fundamental problem in statistics and machine learning.

By Yuha Park, Yongdai Kim

benchmarks

arXiv Machine Learning

Jun 16

Information Leakage Detection through Approximate Bayes-optimal Prediction

arXiv:2401. 14283v4 Announce Type: replace-cross Abstract: In today's data-driven world, the proliferation of publicly available information raises security concerns due to the information leakage (IL) problem.

By Pritha Gupta, Marcel Wever, Eyke H\"ullermeier

benchmarks

arXiv Machine Learning

Jun 16

Using Reinforcement Learning to Optimize the Global and Local Crossing Number

arXiv:2509. 06108v2 Announce Type: replace-cross Abstract: Graph drawing concerns the algorithmic visualization of graphs.

By Timo Brand, Henry F\"orster, Stephen Kobourov, Daniel Kohrt, Robin Schukrafft, Markus Wallinger, Johannes Zink

agents reinforcement-learning benchmarks

arXiv AI

Jun 16

P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

arXiv:2606. 16753v1 Announce Type: cross Abstract: As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use.

By Rafael Ferreira, In\^es Vieira, In\^es Calvo, James Furtado, Iago Paulo, Diogo Tavares, Diogo Gl\'oria-Silva, David Semedo, Jo\~ao Magalh\~aes

llms benchmarks safety

arXiv AI

Jun 16

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

arXiv:2606. 16825v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory.

By Martin Jaggi

llms efficiency benchmarks

Get the daily briefing