Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 7

Stochastic Penalty-Barrier Method for Constrained Machine Learning

arXiv:2605.18618v3 Announce Type: replace-cross Abstract: Constrained Machine Learning (CML) enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain know...

By Adam Bos\'ak, Andrii Kliachkin, Gilles Bareilles, Allen Gehret, Allahkaram Shafiei, Jana Lep\v{s}ov\'a, Jakub Mare\v{c}ek

benchmarks safety

arXiv AI

Oct 7

Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

arXiv:2605.30557v2 Announce Type: replace-cross Abstract: Spatial reasoning benchmarks typically evaluate whether vision-language models can derive the correct answer from a visual observation. Yet i...

By Yue Zhang, Zun Wang, Han Lin, Yonatan Bitton, Idan Szpektor, Mohit Bansal

llms fine-tuning multimodal benchmarks

arXiv AI

Oct 7

Variational-Ising-Attention:Tailored Attention Matters for Science

arXiv:2607.23634v2 Announce Type: replace-cross Abstract: Attention enables context modeling via query-key scoring with softmax normalization. Driven by industrial long-context demands, mainstream re...

By Rui Wang

efficiency benchmarks

arXiv AI

Oct 7

More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

arXiv:2610.04753v2 Announce Type: replace-cross Abstract: Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Spa...

By Noam Elata, Itay Lamprecht, Mikey Shechter, Daniel Ohayon, Itay Hubara, Daniel Soudry

llms fine-tuning efficiency benchmarks

arXiv AI

Oct 7

Cross-Lingual Activation Steering for Multilingual Language Models

arXiv:2601.16390v2 Announce Type: replace-cross Abstract: Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant la...

By Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar

llms benchmarks safety

arXiv Machine Learning

Oct 7

Uncertainty in Representation Learning on Knowledge Graphs

arXiv:2610.06974v1 Announce Type: new Abstract: Knowledge graph embedding (KGE) methods represent entities and predicates in continuous vector spaces to infer missing knowledge. Despite strong benchm...

By Yuqicheng Zhu

rag benchmarks

arXiv Machine Learning

Oct 7

STOCK-JEPA: Prior-Anchored Latent Revision Representation Learning in Equity Markets

arXiv:2610.07006v1 Announce Type: new Abstract: Learning effective representations helps characterize the structure and dynamics of equity markets from financial data with a low signal-to-noise ratio...

By Yizhi Luo, Jiahe Yi, Jianhui Zhang, Shuo Sun

rag diffusion benchmarks

arXiv Machine Learning

Oct 7

SoloQ: Calibration-Free Quantization for Diffusion Language Models

arXiv:2610.07121v1 Announce Type: new Abstract: Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based...

By Donghyun Lee, Arkapravo Ghosh, Varun Manjunath, Bumjoon Kyle Rhee, Hyunho Kook, Shiting Xiao, Youngeun Kim, Priyadarshini Panda

llms diffusion efficiency benchmarks

arXiv Machine Learning

Oct 7

Benchmarking Time Series Foundation Models for Load Forecasting Under Covariate Uncertainty

arXiv:2610.07232v1 Announce Type: new Abstract: Accurate short-term load forecasting (STLF) is essential for the reliable and efficient operation of modern power systems. While time series foundation...

By Tomas Kaljevic, Ivan Arzola, Yu Zhang

fine-tuning benchmarks

arXiv Machine Learning

Oct 7

Structuring MoE Expert Selection for Agentic Reinforcement Learning

arXiv:2610.07332v1 Announce Type: new Abstract: Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structur...

By Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Oct 7

Multigroup Fairness and Omniprediction: Separations and Equivalences

arXiv:2610.07374v1 Announce Type: new Abstract: Omniprediction is a learning guarantee which requires a single predictor to be competitive relative to the best hypothesis from a benchmark class for a...

By S\'ilvia Casacuberta, Parikshit Gopalan, Varun Kanade, Omer Reingold, Konstantinos Stavropoulos, Pranay Tankala

benchmarks safety

arXiv Machine Learning

Oct 7

Benchmarking Label-Revealed Online Updates for EEG BCI Decoding

arXiv:2610.07420v1 Announce Type: new Abstract: Electroencephalography (EEG) signals drift over time, which can cause static brain-computer interface (BCI) models to degrade in practice. We present a...

By Bogdan Kozyrskiy, Artem Grachev, Abraham I. Camelo Guerrero

benchmarks

arXiv Machine Learning

Oct 7

Interpretable Hypergraph Learning via Neural Additive Models

arXiv:2610.07458v1 Announce Type: new Abstract: Hypergraphs offer a natural framework for modeling networked data, where dependencies among entities are governed by higher-order interactions. While h...

By Shihan Feng, Xin Zheng, Shiyi Yang, Ren Wang, Chudi Zhong, Can Chen

benchmarks safety

arXiv Machine Learning

Oct 7

Robust Importance Sampling for Rare Events via Constrained Gaussian Mixtures

arXiv:2610.07485v1 Announce Type: new Abstract: We study estimating rare-event probabilities $I = \mathbb{P}(g(\mathbf{X}) > \gamma)$ with $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\...

By Pawe{\l} Lorek, Rafa{\l} Nowak, Rafa{\l} Topolnicki, Tomasz Trzci\'nski, Maciej Zi\k{e}ba

multimodal benchmarks

arXiv Machine Learning

Oct 7

Adaptive Model Inversion Attacks Generalize a Privacy-Robustness Tradeoff

arXiv:2610.07677v1 Announce Type: new Abstract: In this paper, we show that standard evaluations of high-resolution Model Inversion Attacks (MIAs) significantly underestimate training-data privacy le...

By Shailen Smith, Rasmus Torp, Adam Breuer

benchmarks safety

arXiv Machine Learning

Oct 7

Extending Pathwise Gradients to Discrete Random Variables via Finite-Order Relaxation

arXiv:2610.07786v1 Announce Type: new Abstract: Pathwise gradients are preferred for continuous random variables because they are unbiased, low variance, and work with a single sample. For discrete v...

By Donghan He, Luhuan Wu

benchmarks safety

arXiv Machine Learning

Oct 7

TTNet: Multi-Task Deep Learning for Table Tennis Player Analysis with Smart Racket

arXiv:2610.07823v1 Announce Type: new Abstract: The AI CUP 2025 Precise Analysis of Table Tennis Smart Racket Data Competition introduced smart table tennis rackets that collect extensive player swin...

By Ko-Hsun Chen, Xiang-Wei Ke, Hsien-Cheng Huang, Shang-Kuan Chen

benchmarks

arXiv Machine Learning

Oct 7

Retrieval Is Not Enough: Refreshing Memory for Frozen Time-Series Forecasters

arXiv:2610.07834v1 Announce Type: new Abstract: Retrieval-augmented time-series forecasting uses the continuations of historical segments similar to the current context as references for a forecaster...

By Chao He, Jianyu Xu, Xinyi Guo, Ruiqi Liu, Haobin Ding, Ruiqi He, Dongqing Song

rag benchmarks safety

arXiv Machine Learning

Oct 7

Do Higher-Order Models Win for Higher-Order Reasons? Rethinking Performance Gains in Hypergraph Learning

arXiv:2610.07981v1 Announce Type: new Abstract: Higher-order models (e.g., hypergraph neural networks) often outperform lower-order baselines on hypergraph learning benchmarks, and their advantages a...

By Fanchen Bu, Fan Li, Geon Lee, Sunwoo Kim, Xiaoyang Wang, Renaud Lambiotte, Kijung Shin

benchmarks

arXiv Machine Learning

Oct 7

Structure-Aware Graph Abstention for Reliable Selective Forecasting

arXiv:2610.08322v1 Announce Type: new Abstract: Selective forecasting abstains on high-risk test windows under a retained-coverage budget. Existing gates such as TEM (Brusokas et al., 2025) score eac...

By Jianxiang Xie, Belal Alsinglawi

benchmarks safety

Get the daily briefing