Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning

Oct 7

Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty

arXiv:2610.08578v1 Announce Type: new Abstract: Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising...

By Amir Mohammad Mahfoozi, Zi Yang, Ying Li, Michael Minyi Zhang

llms benchmarks safety

arXiv Machine Learning

Oct 7

Multi-Label Perceptual Bug Detection in Video Games using Deep Learning on Gameplay Footage

arXiv:2610.08593v1 Announce Type: new Abstract: Traditional approaches for automated bug detection in video games, such as manual testing, can be beneficial for the improvement of quality assurance,...

By Nahian Rifaat, Felix Morosov, Loutfouz Zaman

benchmarks

arXiv Machine Learning

Oct 7

Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling

arXiv:2610.08677v1 Announce Type: new Abstract: Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly...

By Nicol\`o Felicioni, Michael Benigni, Maurizio Ferrari Dacrema, Paolo Cremonesi

reinforcement-learning benchmarks

arXiv Machine Learning

Oct 7

FactorBench: A Portfolio-Aware Benchmark for Automated Factor Mining

arXiv:2610.06947v1 Announce Type: cross Abstract: Factor mining seeks to discover signals from financial data that predict future asset returns and guide portfolio construction. Automated factor mini...

By Zhuohan Wang, Carmine Ventre

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Oct 7

Hierarchy-GBP: Accelerating Factor Graph Inference via Abstraction and Recovery

arXiv:2610.06978v1 Announce Type: cross Abstract: Gaussian Belief Propagation (GBP) is a distributed inference algorithm that passes messages in graphical models, making it attractive for scalable sp...

By Yuzhou Cheng, Tom Yates, Ignacio Alzugaray, Danyal Akarca, Pedro A. M. Mediano, Andrew J. Davison

benchmarks

arXiv Machine Learning

Oct 7

Behavioral Cloning Mystery

arXiv:2610.07056v1 Announce Type: cross Abstract: Behavioral cloning (BC), despite its simplicity, exhibits many counterintuitive phenomena in the real world. For example, the performance of BC often...

By Seohong Park, Sergey Levine

rag robotics benchmarks

arXiv Machine Learning

Oct 7

ImpactMat: Continuous Material Estimation for Inverse Impact Sound Rendering

arXiv:2610.07061v1 Announce Type: cross Abstract: Impact sound rendering synthesizes the sound produced when a 3D object is struck, but practical renderers often rely on fixed material presets such a...

By Hyebin Cho, Bumsoo Kim, Joon son Chung

benchmarks

arXiv Machine Learning

Oct 7

Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

arXiv:2610.07094v1 Announce Type: cross Abstract: LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test...

By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid

llms agents efficiency benchmarks

arXiv Machine Learning

Oct 7

BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation

arXiv:2610.07594v1 Announce Type: cross Abstract: Humanoid household manipulation requires the arms to act while the body balances, steps and changes posture. We present BiGym 2.0, an adaptation of B...

By Zexi Zhang, Zecheng Zhu, Zidong Chen, Zulkhuu Tuya, Stephen James

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Oct 7

Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects

arXiv:2610.07755v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and aver...

By Tianxi Li, Jie Ding

llms benchmarks

arXiv Machine Learning

Oct 7

Learned Adaptive Multiresolution Diffusion Imaging

arXiv:2610.07884v1 Announce Type: cross Abstract: Adaptive multiresolution methods reduce representation cost by concentrating fine-scale degrees of freedom where needed, but their tree updates are u...

By Christian Tantardini, Stig Rune Jensen, Roberto Di Remigio Eik{\aa}s, Joakim Henrik Beck

diffusion benchmarks

arXiv Machine Learning

Oct 7

Learning consistent molecular mechanics force fields from first principles

arXiv:2610.08020v1 Announce Type: cross Abstract: Classical force fields (FFs) remain the workhorse for large-scale simulations even as machine-learned interatomic potentials (MLIPs) approach ab init...

By Berkay G\"unes, Leif Seute, Jigyasa Nigam, Frauke Gr\"ater

benchmarks

arXiv Machine Learning

Oct 7

Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

arXiv:2610.08173v1 Announce Type: cross Abstract: Automated Program Repair (APR) with language models is usually evaluated by whether a generated patch passes the test suite, which can hide differenc...

By Anvi Kalpesh Shah, Umamaheswara Sharma B

llms benchmarks

arXiv Machine Learning

Oct 7

Two-Sample Testing via Generative Processes

arXiv:2610.08277v1 Announce Type: cross Abstract: Deciding whether two samples come from the same distribution is a classical problem in statistics, and generative transport offers a new way to appro...

By Eshant English, Kenji Fukumizu, Taiji Suzuki

benchmarks

arXiv Machine Learning

Oct 7

Where Do Two Populations of Persistence Diagrams Differ? Calibrated Local Inference at a Fixed Budget

arXiv:2610.08292v1 Announce Type: cross Abstract: Many two-sample tests for populations of persistence diagrams assess global differences without identifying the regions of the birth-death plane that...

By Pramita Bagchi, Edward Bae, Atish Mitra, Alexander D. Silberman, \v{Z}iga Virk, Sushovan Majhi

benchmarks

arXiv Machine Learning

Oct 7

DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

arXiv:2610.08341v1 Announce Type: cross Abstract: Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redu...

By Shuo Yang, Changbai Li, Linlin Yang, Huobin Tan, Rongyu Chen, Tongfei Chen, Tian Wang, Sheng Xu, Baochang Zhang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Oct 7

Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents

arXiv:2610.08452v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is a widely used approach for grounding large language models (LLMs) in external knowledge. However, configuring...

By Lasse B. Strand, Robert Jakob, Kevin O'Sullivan, Markus Kreft

llms rag agents benchmarks

arXiv Machine Learning

Oct 7

Probabilistic Truly Unordered Rule Sets

arXiv:2401.09918v2 Announce Type: replace Abstract: Rule set learning has recently been frequently revisited because of its interpretability. Existing methods have several shortcomings though. First,...

By Lincen Yang, Matthijs van Leeuwen

benchmarks safety

arXiv Machine Learning

Oct 7

Direct Regret Optimization in Bayesian Optimization

arXiv:2507.06529v2 Announce Type: replace Abstract: Bayesian optimization (BO) is a powerful paradigm for optimizing expensive black-box functions. Traditional BO methods typically rely on separate h...

By Fengxue Zhang, Yuxin Chen

llms benchmarks

arXiv Machine Learning

Oct 7

HARL-A: An Extensible Benchmark Framework for Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLab

arXiv:2510.01264v2 Announce Type: replace Abstract: Progress in adversarial multi-agent reinforcement learning (MARL) for robotics has been hampered by a lack of shared, extensible infrastructure tha...

By Isaac Peterson, Christopher Allred, Jacob Morrey, Mario Harper

agents reinforcement-learning robotics benchmarks safety

Get the daily briefing