Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning

Oct 15

Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

arXiv:2610.19095v1 Announce Type: cross Abstract: Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory...

By Soovadeep Bakshi, Apurba Bose, Aishwarya Bose, Sandip Nandi

llms benchmarks

arXiv Machine Learning

Oct 15

Into the Unknown: Benchmarking AI agents on open scientific research problems

arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper...

By Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck

agents benchmarks

arXiv Machine Learning

Oct 15

ActMOF: Mapping Reticular Synthesis through Active Learning

arXiv:2610.19404v1 Announce Type: cross Abstract: Reticular chemistry allows metal-organic frameworks (MOFs) to be designed from their building units, yet their synthesis remains constrained by spars...

By Hongrui Sheng, Shutian Liang, Reid A. Coyle, Shyam Chand Pal, Peter Walther, Bin Feng, Xinyu Ding, Zhiling Zheng

llms benchmarks

arXiv Machine Learning

Oct 15

Bridging the Topological Divide: Dual-Branch Graph Distillation with Semantic Priors for Robust Cold-Start Recommendation

arXiv:2610.19578v1 Announce Type: cross Abstract: Graph Collaborative Filtering models have achieved remarkable success in recommendation systems. However, they inherently suffer from a catastrophic...

By Yuanfeng Qiu

efficiency multimodal benchmarks safety

arXiv Machine Learning

Oct 15

Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards

arXiv:2610.19868v1 Announce Type: cross Abstract: Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification...

By Tianyu Liu, Davide Belli, Bence Major, Arash Behboodi, Tribhuvanesh Orekondy

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry

arXiv:2610.20350v1 Announce Type: cross Abstract: Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed r...

By Ren Zhuang, Ben Wang, Shuifa Sun

reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

arXiv:2507.14843v5 Announce Type: replace Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...

By Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

llms reinforcement-learning benchmarks

arXiv Machine Learning

Oct 15

A Versatile Variational Quantum Kernel Framework for Non-Trivial Classification

arXiv:2511.10831v3 Announce Type: replace Abstract: Quantum kernel methods are a promising branch of quantum machine learning, but their per- formance across data types and encoding choices remains i...

By Jiang Yuhan, Matthew Otten

benchmarks

arXiv Machine Learning

Oct 15

Deep Time-series Forecasting Needs Kernelized Moment Balancing

arXiv:2602.00717v2 Announce Type: replace Abstract: Deep time-series forecasting can be formulated as a distribution balancing problem aimed at aligning the distribution of the forecasts and ground t...

By Licheng Pan, Haocheng Yang, Shuting He, Han Zheng, Yuzhou Chen, Guangyi Liu, Kangning Zhang, Zhixuan Chu, Yuan Lu, Hao Wang

benchmarks

arXiv Machine Learning

Oct 15

Causally-Constrained Probabilistic Forecasting for Time-Series Anomaly Detection

arXiv:2604.17998v2 Announce Type: replace Abstract: Anomaly detection in multivariate time series is a central challenge in industrial monitoring, as failures frequently arise from complex temporal d...

By Pooyan Khosravinia, Jo\~ao Gama, Bruno Veloso

llms benchmarks safety

arXiv Machine Learning

Oct 15

Generative Actor-Critic with Soft Bridge Policies

arXiv:2605.08733v2 Announce Type: replace Abstract: Expressive generative policies such as diffusion and flow models are appealing for maximum-entropy (MaxEnt) online reinforcement learning because o...

By Ke He, Le He, Shunpu Tang, Yafei Wang, Lisheng Fan

diffusion reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Oct 15

Privacy-Preserving GNN Inference on Financial Transaction Graphs via Homomorphic Encryption

arXiv:2610.18054v2 Announce Type: replace-cross Abstract: Anti-money laundering (AML) in financial networks requires modeling transaction patterns that span multiple institutions. In practice, howeve...

By H. \c{C}a\u{g}r{\i} Bilgi, Kemal Mutluergil, Ceren Y{\i}ld{\i}r{\i}m, Kubilay Atasu, Erkay Savas

efficiency benchmarks safety

arXiv Machine Learning

Oct 15

TabFORGE: A Tabular Foundation Model for Generative Modelling

arXiv:2605.09424v2 Announce Type: replace Abstract: Generative modelling is a demanding test of foundation models, because it requires robust, holistic representation learning for a given data modali...

By Xiangjian Jiang, Mingxuan Liu, Nikola Simidjievski, Tassilo Klein, Mateja Jamnik

llms rag diffusion benchmarks safety data-engineering

arXiv Machine Learning

Oct 15

LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection

arXiv:2605.10980v2 Announce Type: replace Abstract: Diffusion Language Models (dLLMs) have garnered significant attention for their potential in highly parallel processing. The parallel capabilities...

By Haohui Zhang, Zhiye Wang, Xiaoying Gan, Xinbing Wang, Bo Jiang

llms diffusion efficiency benchmarks safety

arXiv Machine Learning

Oct 15

tidyHEBO: Improving the Robustness of Heteroskedastic Evolutionary Bayesian Optimization

arXiv:2607.10669v3 Announce Type: replace Abstract: Bayesian optimization is widely used for expensive black-box optimization problems. While methodological advances often focus on individual compone...

By L. A. Zhukov, E. V. Shaburova, D. V. Antonets

benchmarks

arXiv Machine Learning

Oct 15

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

arXiv:2311.04902v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurt...

By Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma, Zhiqiang Shen

llms computer-vision efficiency benchmarks

arXiv Machine Learning

Oct 15

Sharp Statistical Thresholds for Attributed Gaussian Graph Matching

arXiv:2603.23305v2 Announce Type: replace-cross Abstract: We investigate the problem of matching networks where both edge weights and node features are correlated following a specific node correspond...

By Mohammad Hassan Ahmad Yarandi, Luca Ganassali

benchmarks safety

arXiv Machine Learning

Oct 15

Attention Transfer Is Not Universally Effective for Vision Transformers

arXiv:2605.07191v2 Announce Type: replace-cross Abstract: A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT)...

By Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Peng Hu, Chen Gong, Xi Peng, Hongyuan Zhu

llms computer-vision benchmarks

arXiv Machine Learning

Oct 15

Memory by Design: Bayesian Layers for Sequence Modeling

arXiv:2605.31163v5 Announce Type: replace-cross Abstract: We introduce the \emph{design-model framework}: a way to derive efficient recurrent sequence maps from explicit assumptions about memory. A d...

By Matthew Dowling, Hyungju Jeon, Cristina Savin, Il Memming Park

benchmarks

arXiv Computation and Language

Oct 15

Can LLMs Access their World Knowledge for Event Prediction?

arXiv:2610.19021v1 Announce Type: new Abstract: Recent work has questioned whether large language models (LLMs) can be useful for temporal event prediction. Despite vast pretrained world knowledge, L...

By Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi, Nikita Dvornik, Alexander Pashevich

llms benchmarks safety

Get the daily briefing