Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 2

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

arXiv:2609.33153v2 Announce Type: replace-cross Abstract: Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative r...

By Shuyang Zhang

llms agents benchmarks

arXiv AI

Oct 2

Talked Out of the Truth: Sycophancy in the Reasoning Chains of Multimodal Models

arXiv:2608.28623v2 Announce Type: replace-cross Abstract: Large multimodal reasoning models (LMRMs) are increasingly capable, largely through generating explicit chain-of-thought reasoning before ans...

By Mahir Numayeer Islam, Gakuto Okuyama, Nikolaus Siauw, Shivank Garg, Madhur Panwar, Vasu Sharma

llms multimodal benchmarks

arXiv Machine Learning

Oct 2

A Data-Free Physics-Informed Neural Operator for Level-Set Interface Advection

arXiv:2609.38195v1 Announce Type: new Abstract: Operators for interfacial problems are trained on reference solutions produced by the solver they are intended to replace. This work develops a data-fr...

By Muhammad Akbar Khan

benchmarks

arXiv Machine Learning

Oct 2

Continual Learning of Dynamical Systems in Recurrent Neural Networks through Recyclable Unit Gating

arXiv:2609.38356v1 Announce Type: new Abstract: Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continu...

By Sima Hashemi, Daniel Durstewitz, Georgia Koppe

agents benchmarks

arXiv Machine Learning

Oct 2

Graph Anomaly Detection as Finite-Horizon Control: Training-Free Scoring via Empirical Bayes

arXiv:2609.38424v1 Announce Type: new Abstract: Node-level graph anomaly detection (GAD) identifies nodes whose attributes and interactions deviate from dominant graph regularities. Existing GAD mode...

By Fred Xu, Thomas Markovich, Florence Regol, Yizhou Sun

benchmarks

arXiv Machine Learning

Oct 2

Geometry-physics confounding impairs PDE learning across varying domains

arXiv:2609.38623v1 Announce Type: new Abstract: Learning partial differential equation (PDE) dynamics across varying domains is central to predictive modelling and data-driven discovery of governing...

By Yinghao Cheng, Gengxiang Chen, Xu Liu, Qinglu Meng, Yixin Jing, Xiangguo Tang, Wenping Mou, Lihui Wang, Yingguang Li

benchmarks

arXiv Machine Learning

Oct 2

SCORE-LM: State-Space Radar Representations with Language Models for Fault Diagnosis

arXiv:2609.38980v1 Announce Type: new Abstract: Radar hardware faults threaten automated perception, motivating accurate, compact diagnosis and understandable maintenance guidance. We introduce SCORE...

By Mainak Mallick, Seung-Kyum Choi

llms benchmarks

arXiv Machine Learning

Oct 2

Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing

arXiv:2609.39067v1 Announce Type: new Abstract: Elastic Compute Cloud (EC2) Spot is 60% to 90% cheaper than On-Demand but can be reclaimed on just a 2-minute notice; for expensive multi-node training...

By Angshuman Chakravertty, MD Rayyan

benchmarks

arXiv Machine Learning

Oct 2

SparseEngine: Sparse-First Inference Engine

arXiv:2609.39068v1 Announce Type: new Abstract: Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these...

By Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu

llms agents efficiency benchmarks

arXiv Machine Learning

Oct 2

Whitening Improves Robustness to Spurious Correlations in Linear Probes

arXiv:2609.39177v1 Announce Type: new Abstract: Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear p...

By Floris Holstege, Bram Wouters, Noud van Giersbergen, Cees Diks

benchmarks

arXiv Machine Learning

Oct 2

QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs

arXiv:2609.39223v2 Announce Type: new Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...

By Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben Athiwaratkun

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Oct 2

Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventions

arXiv:2609.39243v1 Announce Type: new Abstract: Causal interventions such as activation patching and distributed alignment search (DAS) are the main tool for making mechanistic claims about neural ne...

By Beiming Liu, Minjie Chen

llms benchmarks safety

arXiv Machine Learning

Oct 2

ReTaCo: Residual-Target Control for On-Policy Distillation

arXiv:2609.39275v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own generated prefixes with token-level teacher feedback, but transmitting or storing the teacher'...

By Zixiang Ni, Zhuo Hu, Renjie Cao, Weijie Ren, Binqin Shi, Weijia Zhang, Shuheng Cao, Zhicheng Shi, Zhenhao Zhang, Haomin Wen, Zhiyuan Hu

efficiency benchmarks

arXiv Machine Learning

Oct 2

When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events

arXiv:2609.39386v1 Announce Type: new Abstract: Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on whi...

By Daniel Schoess, Florian von Wangenheim

benchmarks

arXiv Machine Learning

Oct 2

Learning Reliable GUI Agents under Imperfect Priors

arXiv:2609.39547v1 Announce Type: new Abstract: GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GU...

By Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan, Changqiao Wu, Yuanfa Li, Pengzhi Gao, Wei Liu, Jian Luan, Heng Qu, Yunpeng Song, Zhongmin Cai

llms rag agents multimodal benchmarks

arXiv Machine Learning

Oct 2

Self-Repulsive Sampling for Diffusion Language Models

arXiv:2609.39560v1 Announce Type: new Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional...

By Michael Helcig, Martin Jaggi

llms diffusion benchmarks

arXiv Machine Learning

Oct 2

MIND: Marginal-Invariant Neural Dependency Diffusion for Mixed-Type Tabular Generation

arXiv:2609.39628v1 Announce Type: new Abstract: This paper proposes MIND, a marginal-invariant neural dependency diffusion model for mixed-type tabular data. MIND does not directly learn the joint di...

By Pengfei Li, Mohammad Khalil

diffusion benchmarks

arXiv Machine Learning

Oct 2

NodeGround: A Node Classification Benchmark in the Graph Foundation Model Era

arXiv:2609.39673v1 Announce Type: new Abstract: Can a pretrained graph model replace training and tuning a separate predictor for each dataset? Answering this requires evaluating prediction quality a...

By Jinmo Lee, Dooho Lee, Minho Jeong, Jaemin Yoo

benchmarks

arXiv Machine Learning

Oct 2

A library for differentiable signal processing and machine learning on the sphere

arXiv:2609.39737v1 Announce Type: new Abstract: The two-dimensional sphere embedded in three-dimensional Euclidean space S2, plays a central role in a variety of scientific and engineering domains, i...

By Thorsten Kurth, Max Rietmann, Mauro Bisson, Andrea Paris, Alberto Carpentieri, Jean Kossaifi, Anima Anandkumar, Christian Hundt, Boris Bonev

llms benchmarks

arXiv Machine Learning

Oct 2

Stable Transformers for Graph Generation

arXiv:2609.39739v1 Announce Type: new Abstract: Graph generative models increasingly rely on Graph Transformers (GT) to capture complex dependencies among nodes and edges. While deeper architectures...

By Luca Miglior, Alessio Gravina, Davide Bacciu

llms benchmarks

Get the daily briefing