Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,767 stories · RSS feed

arXiv Machine Learning

Jun 25

Learning Structural Hardness for Combinatorial Auctions: Instance-Dependent Algorithm Selection via Graph Neural Networks

arXiv:2602. 14772v2 Announce Type: replace Abstract: The Winner Determination Problem (WDP) in combinatorial auctions is NP-hard, and no existing method reliably predicts which instances will defeat fast greedy heuristics.

By Sungwoo Kang

benchmarks safety

arXiv Machine Learning

Jun 25

Consistent Zero-Shot Imitation with Contrastive Goal Inference

arXiv:2510. 17059v2 Announce Type: replace Abstract: Zero-shot imitation learning requires an agent to reproduce expert behavior from a single demonstration without additional environment interaction or gradient updates at test time.

By Kathryn Wantlin, Chongyi Zheng, Benjamin Eysenbach

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

TurboMPC: Fast, Scalable, and Differentiable Model Predictive Control on the GPU

arXiv:2606. 24039v1 Announce Type: cross Abstract: Robotics increasingly relies on GPUs for parallel simulation, large-scale learning, and neural-network inference.

By Gabriel Bravo-Palacios, Jianghan Zhang, Zachary Pestrikov, Brian Plancher, Thomas Lew

reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jun 25

Laplace--Fisher Gate Identities for Optimal Matrix-Gated Blended Score Estimation

arXiv:2606. 25169v1 Announce Type: cross Abstract: Sampling from an unnormalized target by reversing an Ornstein--Uhlenbeck diffusion requires the score of each noise-perturbed marginal.

By Alois Duston, Tan Bui Tanh

diffusion benchmarks

arXiv Machine Learning

Jun 25

Generating Input Distributions for Explaining Portfolio Optimization Pipelines

arXiv:2606. 25808v1 Announce Type: cross Abstract: We propose a predict-optimize-explain framework that uses gradient-based sample generation to interpret various portfolio models by identifying macroeconomic conditions that induce specified portfolio outcomes.

By Batuhan Ata\c{s}, Nur\c{s}en Ayd{\i}n, E. Mehmet K{\i}ral, \c{S}. \.Ilker Birbil

benchmarks

arXiv Machine Learning

Jun 25

OmegAMP: Targeted AMP Discovery via Biologically Informed Generation

arXiv:2504. 17247v3 Announce Type: replace Abstract: Deep learning-based antimicrobial peptide (AMP) discovery faces critical challenges such as limited controllability, lack of representations that efficiently model antimicrobial properties, and low experimental hit rates.

By Diogo Soares, Leon Hetzel, Paulina Szymczak, Marcelo Der Torossian Torres, Johanna Sommer, Cesar de la Fuente-Nunez, Fabian Theis, Stephan G\"unnemann, Ewa Szczurek

diffusion benchmarks

arXiv Machine Learning

Jun 25

Stable-Shift: Biologically Structured Prediction of Transcriptional Responses to Unseen Gene Perturbations

arXiv:2606. 24940v1 Announce Type: cross Abstract: Predicting transcriptional responses to genetic perturbations could reduce the experimental burden of functional genomics, but extrapolation to genes that were never perturbed during training remains difficult.

By Sajib Acharjee Dip, Liqing Zhang

benchmarks

arXiv Machine Learning

Jun 25

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

arXiv:2606. 25519v1 Announce Type: cross Abstract: Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency.

By Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang

llms agents nlp efficiency benchmarks

arXiv Machine Learning

Jun 25

LLM-ACES: Closed-Loop Discovery of Dynamical Systems with LLM-Guided Adaptive Search

arXiv:2606. 25039v1 Announce Type: new Abstract: Recovering governing Ordinary Differential Equations (ODEs) from data is a central challenge in modeling dynamical systems across scientific domains.

By Nikhil Abhyankar, Sha Li, Sanchit Kabra, Naren Ramakrishnan, Yulia Gel, Chandan K. Reddy

llms benchmarks

arXiv Machine Learning

Jun 25

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

arXiv:2511. 11421v2 Announce Type: replace-cross Abstract: Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.

By Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang, Han-Jia Ye, De-Chuan Zhan

llms multimodal benchmarks

arXiv Machine Learning

Jun 25

Frequency Domain Reservoir Computing

arXiv:2606. 24969v1 Announce Type: new Abstract: While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states.

By Klaus Schertler, Xiomara Runge, Andrea Ceni, David Kappel, Claudio Gallicchio

llms rag benchmarks

arXiv Machine Learning

Jun 25

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

arXiv:2606. 26079v1 Announce Type: cross Abstract: Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines.

By Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 25

Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

arXiv:2606. 24965v1 Announce Type: cross Abstract: Reasoning about relational structures remains a significant challenge for neural models, particularly when they must systematically apply learned knowledge to problem instances that are harder than those seen in training.

By Anirban Das, Joanne Boisson, Irtaza Khalid, Sumita Garai, Steven Schockaert

llms agents benchmarks

arXiv Machine Learning

Jun 25

Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents

arXiv:2606. 26080v1 Announce Type: new Abstract: Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale.

By Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick, Sharon Li

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 25

Agentic evolution of physically constrained foundation models

arXiv:2606. 25532v1 Announce Type: cross Abstract: Artificial intelligence increasingly drives automated scientific discovery, yet contemporary generalist agents lack physical grounding, frequently hallucinating hardware-incompatible designs.

By Jiangwei Zhang, Wen Sun, Chong Wang, Shiyao Li, Cheng Che, Chunjing Han, Dan Meng, Jian Yang, Yu Wang, Rui Hou

agents benchmarks

arXiv Machine Learning

Jun 25

When Multi-Sensor Fusion Fails to Generalize: Cattle Posture Classification Under Animal-Level and Temporal Distribution Shift

arXiv:2606. 24986v1 Announce Type: new Abstract: Automated cattle posture-classification systems frequently report near-perfect accuracy, yet their robustness under realistic deployment conditions remains largely unknown.

By Leutrim Uka, Severino Pinto, Gundula Hoffmann, Marina M. -C. H\"ohne

multimodal benchmarks

arXiv Machine Learning

Jun 25

Do Thinking Tokens Help with Safety?

arXiv:2606. 25013v1 Announce Type: new Abstract: Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts.

By Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

llms benchmarks safety

arXiv Machine Learning

Jun 25

MacroLens: A Multi-Task Benchmark for Contextual Financial Reasoning under Macroeconomic Scenarios

arXiv:2606. 24950v1 Announce Type: new Abstract: Financial decision-making is contextual: forecasting prices, valuing companies, and assessing event exposure weigh price history, accounting fundamentals, macroeconomic regime, and contemporaneous text.

By Patara Trirat, Jin Myung Kwak, Jay Heo, Heejun Lee, Sung Ju Hwang

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 25

TopoCast: A Topological Fidelity Framework for Evaluating Transformer-Based Time Series Forecasting

arXiv:2606. 25439v1 Announce Type: new Abstract: Deep learning-based models have achieved state-of-the-art performance in Time Series Forecasting (TSF), yet their evaluation remains dominated by pointwise error metrics such as Mean Squared Error (MSE), which quantify numerical accuracy but overlook structural properties of the forecast signal, including recurrent dynamics, oscillatory behavior, and phase alignment.

By Sandeepa Weerasekara, Sandareka Wickramanayake

llms rag benchmarks safety

arXiv Machine Learning

Jun 25

Are Tabular Foundation Models Robust to Realistic Query Distribution Shifts in Microbiome Data?

arXiv:2606. 24995v1 Announce Type: new Abstract: Tabular foundation models (TFMs) achieve strong performance on microbiome abundance data, yet their robustness under realistic distribution shift remains poorly characterized.

By Giulia Perciballi, Ahmad Fall, Federica Granese, Edi Prifti, Jean-Daniel Zucker

llms benchmarks

Get the daily briefing