Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning

Sep 15

Geometric Signatures of Conceptual Reorganization: A Counterfactual Embedding Framework for Detecting Scientific Revolutions

arXiv:2609.14917v1 Announce Type: cross Abstract: We introduce document embedding geometry as a quantitative observable of conceptual reorganization and develop a counterfactual ablation framework fo...

By Dimitris Ntounis, Ariel Schwartzman, Chris Chafe, Thomas A. Ryckman

llms rag benchmarks

arXiv Machine Learning

Sep 15

ABSOL: Aggregated Bayesian Subsampling Orchestrated with LLMs

arXiv:2609.15007v1 Announce Type: cross Abstract: Large language models are increasingly used as natural-language interfaces to structured data, yet they remain unreliable when answers require consis...

By Jackson Hassell, Chen Shen, Estevam Hruschka

llms benchmarks

arXiv Machine Learning

Sep 15

Temperature Fragility and the Conditional Benefits of Truncation Sampling

arXiv:2609.15476v1 Announce Type: cross Abstract: Large language models generate text by sampling each token from a predicted distribution, and a temperature parameter sets how far the draw strays fr...

By Francesco La Rosa

llms benchmarks

arXiv Machine Learning

Sep 15

Proportional-Fair Resource Allocation and Dual-Threshold Early-Exit Inference for Secure Cooperative Multi-Layer Edge Intelligence

arXiv:2609.15847v1 Announce Type: cross Abstract: This paper proposes FREDI (Fair Resource Allocation for Edge Dual-Threshold Inference), a secure wireless edge-intelligence framework for event-trigg...

By Thai T. Vu, John Le, Tu N. Nguyen, Jun Shen, Quang Vinh Duong, Ha Nguyen

benchmarks safety

arXiv Machine Learning

Sep 15

N$^2$: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completion

arXiv:2506.04166v3 Announce Type: replace Abstract: Nearest neighbor (NN) methods have re-emerged as competitive tools for matrix completion, offering strong empirical performance and recent theoreti...

By Caleb Chin, Aashish Khubchandani, Harshvardhan Maskara, Kyuseong Choi, Jacob Feitelberg, Albert Gong, Manit Paul, Tathagata Sadhukhan, Dwaipayan Saha, Anish Agarwal, Raaz Dwivedi

llms benchmarks

arXiv Machine Learning

Sep 15

Architecture--Optimization Co-Design for Physics-Informed Neural Networks via Layer-wise Coordinate Adaptation and Gradient Conflict Resolution

arXiv:2601.12971v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) can be limited by coordinate representations and conflicting gradients from heterogeneous physical constra...

By Pancheng Niu, Jun Guo, Qiaolin He, Yongming Chen, Yanchao Shi

benchmarks

arXiv Machine Learning

Sep 15

Size Transferability of Graph Transformers with Convolutional Positional Encodings

arXiv:2602.15239v3 Announce Type: replace Abstract: Transformers have achieved remarkable success across domains, motivating the rise of Graph Transformers (GTs) as attention-based architectures for...

By Javier Porras-Valenzuela, Zhiyang Wang, Teresa Shang, Yusu Wang, Alejandro Ribeiro

llms benchmarks

arXiv Machine Learning

Sep 15

Running the Gauntlet: Challenging Agentic Tasks

arXiv:2606.14397v4 Announce Type: replace Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capab...

By Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel, Michal Zakrzewski, Sebastian Montagna, Damian Rynczak, Shreyansh Padarha, Kumail Alhamoud, Zihao Fu, William Lugoloobi, Kai Rawal, Hanna Yershova, Taras Rumezhak, Guohao Li, Fazl Barez, Baoyuan Wu, Arkadiusz Drohomirecki, Yarin Gal, Chris Russell, Christopher Summerfield, Adam Mahdi, Volodymyr Karpiv, Philip Torr, Adel Bibi

agents benchmarks

arXiv Machine Learning

Sep 15

From Linear to Linearizable Optimization: A Novel Framework with Applications to Stationary and Non-stationary DR-submodular Optimization

arXiv:2405.00065v4 Announce Type: replace-cross Abstract: This paper introduces the notion of upper-linearizable/quadratizable functions, a class that extends concavity and DR-submodularity in variou...

By Mohammad Pedramfar, Vaneet Aggarwal

reinforcement-learning benchmarks

arXiv Machine Learning

Sep 15

What resources are needed for optimal learning of bosonic Gaussian states?

arXiv:2603.18136v2 Announce Type: replace-cross Abstract: Continuous-variable systems enable key quantum technologies in computation, communication, and sensing. Bosonic Gaussian states emerge natura...

By Senrui Chen, Francesco Anna Mele, Marco Fanizza, Alfred Li, Zachary Mann, Hsin-Yuan Huang, Yanbei Chen, John Preskill

benchmarks