Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

20,687 stories · RSS feed

arXiv Machine Learning

Jun 24

Exact Schur-Sylvester Dimensionality Reductions for Non-Smooth Stochastic Complexity and Manifold Sampling

arXiv:2606. 23867v1 Announce Type: new Abstract: The exact computation of the Normalized Maximum Likelihood (NML) codelength for regular non-smooth estimators (e.

By Trenton Lau, Gary P. T. Choi

benchmarks

arXiv AI

Jun 24

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.

By Zhuoren Ye, Tianyu Wo, Dinghao Xue, Mingming Zhang, Yuchen Teng, Chunming Hu, Renyu Yang

llms benchmarks

arXiv AI

Jun 24

Are Safety Guarantees in Neural Networks Safe? How to Compute Trustworthy Robustness Certifications

arXiv:2606. 23858v1 Announce Type: cross Abstract: A primary challenge in AI safety is the existence of adversarial examples -- slightly distorted inputs that cause a neural network (NN) to misclassify.

By Merkouris Papamichail, Konstantinos Varsos, Giorgos Flouris, Jo\~ao Marques-Silva

benchmarks safety

arXiv AI

Jun 24

Deciphering Fingerprints of 3D Molecular Surfaces for Accurate Epitope Prediction

arXiv:2606. 23830v1 Announce Type: cross Abstract: Molecular surfaces encode the geometric and physicochemical patterns that determine antibody-antigen recognition, central to epitope prediction.

By Fang Wu, Weihao Xuan, Jure Leskovec, Yejin Choi, Li Erran Li

llms benchmarks

arXiv AI

Jun 24

DTT-BSR+: A Generative-Regression Cascade for Music Source Restoration

arXiv:2606. 24127v1 Announce Type: cross Abstract: Music source restoration (MSR) requires jointly addressing source unmixing and the inversion of non-linear production effects.

By Youran Ni, Shihong Tan, Yuzhu Wang, Gongping Huang

benchmarks

arXiv Machine Learning

Jun 24

Flow-Corrected Thompson Sampling for Non-Stationary Contextual Bandits

arXiv:2606. 23933v1 Announce Type: cross Abstract: We study non-stationary linear contextual bandits where the reward model drifts over time, rendering classical contextual bandit algorithms brittle because historical data becomes systematically biased.

By AmirHossein Naghdi, Ali Baheri

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 24

FAIRVAR: Fair Federated Learning via Variance Regularization

arXiv:2508. 12042v3 Announce Type: replace Abstract: Federated learning (FL) allows collaborative training of machine learning models across multiple parties without sharing raw data.

By Zahra Kharaghani, Ali Dadras, Tommy L\"ofstedt

benchmarks safety

arXiv AI

Jun 24

From Spatial to Spectral: An Efficient, Frequency-Guided Feature Representation Learner for Small Object Detection

arXiv:2606. 23825v1 Announce Type: cross Abstract: Efficient small object detection is bottlenecked by the inherent feature scarcity of tiny targets, which is further aggravated by operations of spatial-domain detectors that indiscriminately discard critical high-frequency details.

By Yuhan Rui, Shihan Qiao, Yibin Lou, Mingxi Yu, Yutong Wan, Yanqiao Chen, Dongsheng Hou, Zhen Cao, Athena Zhuoming Zhong, Qi Hao

llms computer-vision benchmarks

arXiv AI

Jun 24

A P\={a}ninian Foundation for Indic Language Processing

arXiv:2606. 24172v1 Announce Type: cross Abstract: More than a billion people communicate in Indic languages, yet the natural language processing infrastructure serving them remains fragmented and underdeveloped.

By Ritwik Banerjee, Lav R. Varshney

nlp benchmarks safety

arXiv AI

Jun 24

Scaling Laws for Task-Specific LLM Distillation

arXiv:2606. 24747v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical.

By Lavinia Ghita, Dhruv Desai, Ioana Boier

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 24

KLip-PPO: A per-sample KL perspective on PPO-Clip

arXiv:2606. 23932v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning.

By Riccardo Colletti, Robin Holzinger

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 24

GRACE: Gated Refinement for Accurate Causal Edge Discovery in High-Dimensional Time Series

arXiv:2606. 23880v1 Announce Type: new Abstract: From climate teleconnections to gene regulation, modern time-series datasets encompass tens or hundreds of interacting variables, making causal discovery increasingly challenging.

By Mohammad Fesanghary, Abhinav Havaldar

benchmarks

arXiv AI

Jun 24

PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models

arXiv:2606. 24388v1 Announce Type: new Abstract: We introduce a large-scale, open-source dataset of pre-generated adversarial attacks for vision-language models (VLMs).

By Simone Gallivanone, Hossein Khodadadi, Mauro Dore, Mauro Medda, Nicola Franco

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 24

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

arXiv:2606. 24391v1 Announce Type: new Abstract: We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base.

By Arnaud Ricci

llms benchmarks safety

arXiv AI

Jun 24

The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs

arXiv:2606. 24460v1 Announce Type: cross Abstract: Commercial large language models bill, scale latency, and budget context per token.

By Olaoye Anthony Somide

llms nlp benchmarks

arXiv AI

Jun 24

MedPCFM: Improving Medical Point Cloud Completion by Integrating Point Transformers and Flow Matching

arXiv:2606. 24433v1 Announce Type: cross Abstract: Medical point cloud completion is important for anatomical reconstruction and downstream clinical workflows, yet generative modeling in this setting remains insufficiently studied.

By Kamil Kwarciak, Marek Wodzinski

llms diffusion efficiency benchmarks

arXiv Machine Learning

Jun 24

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling

arXiv:2606. 24650v1 Announce Type: cross Abstract: We present Harmonic, a hierarchical state space model (SSM) for language modeling.

By Petr Nyoma

llms benchmarks

arXiv AI

Jun 24

SemChunk-C: Semantic Segmentation for C Code

arXiv:2606. 23697v1 Announce Type: cross Abstract: Semantic segmentation of code written in a C-family language remains a challenging problem, due to the language's complex syntax, macro expansion, and irregular structural patterns.

By Boris Nazarov, Darya Frolova, Shaked Leibzirer, Pavel Kisilev

llms rag computer-vision benchmarks

arXiv AI

Jun 24

Entity Resolution via Batched Oracle Queries

arXiv:2606. 24407v1 Announce Type: cross Abstract: We consider an oracle that processes a limited batch of records at a time and clusters those that refer to the same real-world entity.

By Lorenzo Balzotti, Donatella Firmani, Luca Gagliardelli, Giovanni Simonini

benchmarks

arXiv AI

Jun 24

Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs

arXiv:2511. 20892v4 Announce Type: replace Abstract: Large language models (LLMs) often produce incorrect or outdated content after being employed.

By Xuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Haoyu Wang, Yujun Yan, Haifeng Chen, Zhengzhang Chen

llms benchmarks

Get the daily briefing