Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 22

Fast Bayesian Optimization Through Linear Models

The paper introduces fast Bayesian optimization (BO) methods that leverage linear structures in existing BO algorithms, enabling efficient optimization across a wide range of problem sizes—from 10 to over 6000 dimensions. These new methods achieve performance comparable to state‑of‑the‑art BO while being 100 to 1000 times faster, thereby expanding BO’s applicability to problems that were previously too time‑intensive. The authors also analyze the speed‑performance Pareto frontier, showing that many regions remain unfilled by current algorithms.

By Colin Doumont, Henry Moss, Philipp Hennig
arXiv Machine Learning
Oct 22

Scalable Stress-Constrained Topology Optimization: A Super-resolution and Simultaneous Approach Based on Machine Learning

The paper introduces the simultaneous neural topology optimization (SNTO) framework, which uses custom neural networks to jointly represent state, adjoint, and design variables in a single optimization loop. SNTO incorporates a multi-component loss function that balances design objectives, aggregated stress constraints, and potential energy, and employs a hardware‑aware training scheme that partitions the design domain across multiple GPUs to handle millions of degrees of freedom. Validation on 2D benchmarks shows super‑resolution designs with sharper boundaries and lower volume fractions, while large‑scale 3D tests demonstrate scalability to eight million collocation points on eight GPUs.

By Xiangyu Sun, Shirin Hosseinmardi, Ramin Bostanabad
arXiv Machine Learning
Oct 22

Big Claims, Small Margins: Do Reported GraphLLM Gains Survive Simple Baselines?

The paper evaluates whether recent LLM-based node classification methods on text‑attributed graphs truly outperform simple baselines. By tuning four baseline models on the same data splits and comparing them to the best reported LLM results, the authors find that while some gains are modest (e.g., Correct & Smooth trailing by 2.3–2.8 percentage points), several reported improvements remain significant. They also highlight evaluation risks such as test‑set hyperparameter tuning, misuse of test‑pool labels, undisclosed baseline reuse, and inconsistencies across papers.

By Michal Podstawski
arXiv Machine Learning
Oct 22

A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems

The paper investigates automated failure attribution (AFA) in large language model (LLM)-based agentic systems, where multiple agents collaborate on complex tasks. It finds that LLMs contain sufficient internal knowledge to attribute failures but often fail to output the correct step, revealing a knows‑but‑won’t‑say gap. To address this, the authors introduce grafting—a manual intervention that leverages latent model knowledge—showing it consistently outperforms direct prompting across several LLMs and benchmarks.

By Wendy Zheng, Liang Wu, Yinhan He, Mayank Darbari, Jundong Li, Liangjie Hong
arXiv Machine Learning
Oct 22

Local and Sparse Kernel Operator Learning for Incompressible Flows: An Operator Lagrange Formulation

The paper introduces a local and sparse kernel method for learning solution operators of incompressible flows, replacing global output reconstruction with local divergence‑free interpolants and a multiscale kernel that approaches Kolmogorov statistics under refinement. A compactly supported, positive‑semidefinite kernel yields sparse operator regression, with support radius fitted to data and optional affine polynomials capturing global trends. The method employs Operator Lagrange weights to avoid forming local coefficients, achieving accuracy comparable to or better than previous dense methods and neural operators, while offering significant GPU speed‑ups and reduced storage.

By Ramansh Sharma, Matthew Lowery, Houman Owhadi, Varun Shankar
arXiv Machine Learning
Oct 22

Causal world models under physical parameter shifts

The paper investigates how causal modeling can improve the robustness of world models to physical parameter shifts. By freezing a dependency mask derived from paired interventions, the authors reduce out‑of‑distribution RMSE on a synthetic benchmark from 1.769 to 0.697, nearly matching an oracle mask. In a robotic link‑mass extrapolation task, incorporating explicit affine coefficients for mass effects and using a known PD controller yields a consistent predictive gain, raising median joint R² from 0.609 to 0.727 across all runs, while a learned‑torque model competes with the best recurrent world model when out‑of‑range masses are supplied.

By Chu Lin, Yi Sun
arXiv Machine Learning
Oct 22

Shortcut-Resistant CAM Distillation for Long-Tailed Recognition

The paper introduces Shortcut-Resistant CAM Distillation (SRCD), a plug‑and‑play framework that mitigates shortcut learning in long‑tailed recognition tasks. SRCD transfers object‑focused explanations from head classes to tail classes by aggregating Class Activation Maps (CAMs) into a shortcut‑resistant teacher and distilling it to tail‑class CAMs. The authors provide a theoretical analysis of shortcut reliance in CAM space and demonstrate consistent improvements over strong baselines on long‑tailed benchmarks.

By Wenhai Wan, Teng Zhang, Shao-Yuan Li, Xinrui Wang, Qiang-Sheng Hua, Songcan Chen
arXiv Machine Learning
Oct 22

Probabilistic Machine Learning for Short-Term Atmospheric Seeing Forecasting

The study evaluates machine learning models for forecasting atmospheric seeing up to two hours ahead, emphasizing probabilistic forecasting and uncertainty quantification. Gaussian Processes and a normalizing‑flow‑based model (FloTS) are compared against deterministic RNN and LSTM baselines using Maunakea summit data. Results show that deep learning models, particularly LSTM and FloTS, achieve the best predictive accuracy, with FloTS offering flexible and informative uncertainty estimates.

By Mary-Joe Medlej, Rahul Srinivasan, Simon Prunet, Aziz Ziad, Christophe Giordano
arXiv Machine Learning
Oct 22

SCDM: Scalable Causal Discovery in Nonlinear Temporal Systems with Meta-Learning

SCDM is a shared‑parameter meta‑learning framework designed for scalable causal discovery in nonlinear multivariate time series. It treats each target variable as a task, learning a shared temporal predictor and a shared causal‑strength matrix, and updates only a subset of tasks per meta‑episode to accumulate structural evidence without exhaustive optimization. After training, a graph readout converts the predictor into continuous directed causal scores, and a finite‑error recovery principle guarantees graph recovery under certain assumptions.

By Jingbo Wang, Kegeng Tang, Zihao Wang, Shaogang Ren
arXiv Machine Learning
Oct 22

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

arXiv:2610. 28753v1 Announce Type: cross Abstract: Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs).

By Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu
arXiv Machine Learning
Oct 22

Unlocking Spatial Grounding in Flow-Matching Models via SNR Tuning

The paper shows that the signal‑to‑noise ratio (SNR) distribution is key to achieving fine‑grained spatial control in flow‑matching generative models. By training on high‑noise regimes rather than a centered SNR distribution, the authors raise layout adherence from 25.9 % to 80.8 % on a synthetic benchmark, approaching the 83.6 % ceiling of the underlying variational autoencoder. These insights also improve real‑world layout accuracy on COCO, boosting average precision from 1.0 % to 57.3 % without changing the model architecture.

By Martin Thi{\ss}en, Elke Hergenr\"other
arXiv Machine Learning
Oct 22

Rethinking Post-hoc Debiasing of Large Pretrained Models

The paper examines post‑hoc debiasing techniques for large pretrained models like CLIP, noting that all such methods rely on identifying and expressing bias features in text. It shows that this reliance can cause unexpected performance drops, such as on the UrbanCars benchmark where methods underperform a vanilla classifier even with ground‑truth bias labels. The authors propose a new approach that identifies cross‑class training pairs based on frozen encoder geometry and learns a single linear map for debiasing, achieving competitive results without bias annotations or large language models.

By Divyaansh Mertia, Chirag Bhutra, Himanshu Shekhar, Shubham Bagwari, Pratik Mazumder
arXiv Machine Learning
Oct 22

BrainPro: Process-conditioned Pretraining with Topographic Retrieval for Large EEG Models

BrainPro is a process‑conditioned self‑supervised pretraining framework for large EEG models that addresses two underexplored challenges: the underuse of mental‑process cues in masked pretraining and the variability of electrode montages across datasets. It maps dataset‑specific montages to a universal channel‑region template, retrieves spatial filters to create a topographically aligned basis, and learns a shared encoder along with process‑specific branches conditioned on protocol cues. Experiments on nine public BCI benchmarks show that BrainPro outperforms baselines, with ablations indicating that topographic retrieval and process‑conditioned representation learning jointly enhance EEG decoding.

By Yi Ding, Muyun Jiang, Weibang Jiang, Shuailei Zhang, Xinliang Zhou, Chenyu Liu, Shanglin Li, Yong Li, Cuntai Guan
arXiv Machine Learning
Oct 22

Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

The paper introduces an uncertainty‑aware ROC framework and an interval‑based AUC (iAUC) metric for evaluating interval‑valued risk predictions. It constructs two ROC‑style curves yielding “AUC_L” and “AUC_U”, which decompose rankings into confidently correct, ambiguous, and confidently incorrect regions. The framework links interval coverage to theoretical bounds on optimal AUC and is applicable to Bayesian, bootstrap, and ensemble uncertainty methods, with experiments on clinical data and MIMIC‑IV mortality prediction demonstrating its utility for model comparison and risk‑aware selection.

By Yuqi Li, Matthew M. Engelhard
arXiv Computation and Language
Oct 22

norregel: a preregistered benchmark of dated numeric values in Norwegian law

The paper introduces *norregel*, a preregistered benchmark that tests language models on numeric values in Norwegian legal texts, including amounts, percentages, and age limits, each tied to a specific date. The benchmark contains 916 items extracted from Lovdata, allowing answers to be classified as correct, outdated, wrong, or uninterpretable. Two preregistered runs were conducted: the first showed mixed results across five models, while the second, involving eight models, supported the hypothesis that outdated answers are more common for values introduced after a model’s cutoff, though the robustness of this finding varied across sensitivity analyses.

By Eirik Botten Nicolaysen (EcoDeco AS)
arXiv Computation and Language
Oct 22

OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

OSynth is a large-scale synthetic dataset that simulates organizations, providing 570 hours of meeting audio along with related artifacts and a latent ground-truth state. Human raters can only distinguish OSynth meetings from real transcripts 56% of the time when given short excerpts, indicating the dataset’s realism. When tested with state-of-the-art large language models, adding contextual information from OSynth significantly improves performance on entity-sensitive tasks, highlighting both current limitations and the dataset’s potential for advancing contextual meeting understanding.

By Ian Porada, Suraj Tripathi, Chaitanya Animesh, Leda Sari, Chaitanya Narisetty, Austin R. Matthews, Bohan Yu, Ramesh L. Thulasiram, Gregory Sell, Kean Chin, Dmitriy Genzel
arXiv Computation and Language
Oct 22

Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

Policy Central is a deployed system that assists police departments in identifying relationships between sections of their written policy, such as conflicts, redundancy, similarity, or difference. It offers two architectures: Retrieve‑Judge, a retrieve‑then‑classify pipeline that serves as a low‑cost library‑wide health check, and Screen‑Confirm, a selective audit that exhaustively compares reviewer‑chosen documents. On a benchmark of 302 pairs from one department’s 84 policy documents, Screen‑Confirm achieves 0.795 aggregate recall and 0.885 on conflict detection, while Retrieve‑Judge’s recall is 0.457, highlighting the challenge of efficiently finding relevant pairs.

By Holly Baker, Michael Kuo, Kyana Pollas