Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning
Sep 29

Robust Successor Features

The paper introduces robust successor features, a method that extends the successor representation to handle uncertainty in both reward functions and transition kernels within linear Markov Decision Processes. It provides a theoretical bound on Generalized Policy Improvement that quantifies performance loss due to mismatched dynamics, and demonstrates the approach on grid-based benchmarks against prior methods that consider only reward or transition differences.

By Erik Nikulski, Yamen Habib, Vicen\c{c} Gomez, Anders Jonsson, Rub\'en Moreno-Bote, Javier Segovia-Aguas
arXiv Machine Learning
Sep 29

Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.

By Jo\~ao Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno, Thiago Costa Rizuti da Rocha
arXiv Machine Learning
Sep 29

Improving Molecular-Morphology Contrastive Pretraining using Deep-Learning-based Morphology Profiles

The paper presents MoCoP v2, an enhanced contrastive pretraining method that aligns small molecule embeddings with deep‑learning‑derived cell morphology profiles. By replacing CellProfiler fingerprints with richer image‑encoded features, the new embeddings better capture how molecules alter cell morphology, leading to improved QSAR, toxicity, ADME, and activity predictions. Performance scales log‑linearly with training data size, indicating further gains with larger datasets.

By Jie Li, Kathryn E. Kirchoff, Dante A. Pertusi, Zhizhuo Zhang
arXiv Machine Learning
Sep 29

GyroNovo: Error-Guided Fragment Imputation with Mass-Aware Attention for \textit{De Novo} Peptide Sequencing

GyroNovo is a new framework for de novo peptide sequencing that improves fragment imputation by guiding the process with decoder errors observed during training. It introduces mass-aware attention using rotary embeddings to encode pairwise mass differences between spectral peaks, and creates easy and hard augmented views of spectra to train the decoder under varying corruption levels. Experiments on NovoBench demonstrate significant gains, with about 9 percentage points higher peptide-level precision and 7 percentage points higher amino-acid-level precision compared to the state-of-the-art baseline.

By Abdellah El Mekki, Laks V. S. Lakshmanan, Muhammad Abdul-Mageed
arXiv Machine Learning
Sep 29

PixSim: a calibrated open-source simulator of instant-payment fraud, recovery and interdiction under analyst capacity constraints

PixSim is an open‑source simulator that models Brazil’s Pix instant‑payment system, incorporating irreversible settlement, a regulated recovery mechanism (MED), downstream fund dispersal, and capacity‑constrained analyst review. Calibrated to Banco Central do Brasil data, the model reproduces the 2025 recovery rate within 0.006 and its decomposition within 0.02, and projects future performance under various policy scenarios. Benchmark tests show how staffing levels, alert expiry rules, and scorer quality affect recovery outcomes and false‑block harm.

By Bashir Zeimarani, Alireza Khatib, Somayeh Mousavinasr, Carlos Maur\'icio Serodio Figueiredo
arXiv Machine Learning
Sep 29

Gradient Surgery for Physics-Informed Neural Networks

The paper introduces PAM-GS, a physics-aware gradient surgery technique for Physics-Informed Neural Networks (PINNs). It addresses the highly imbalanced multi-task optimisation problem in PINNs by adaptively mitigating task interference based on observed gradient conflicts. Experiments on four PDE benchmarks show that PAM-GS achieves competitive solution accuracy while maintaining strong task-balanced performance, outperforming existing methods on most problems.

By Thomas Borsani, Giuseppe Di Fatta
arXiv Machine Learning
Sep 29

Learning Hierarchical Causal Representations of the Effects of Forcings on Temperature in Climate Models

The paper presents a hierarchical causal representation learning framework that models both internal climate variability and forced responses in sea surface temperature fields from a global climate model. By training on future climate change scenarios, the method accurately predicts long‑term global mean and regional temperature evolution and reproduces realistic responses to perturbations in greenhouse gas and aerosol concentrations on unseen scenarios. This demonstrates the potential of causal representation learning to improve climate model emulation.

By Shan Zhao, Ilija Trajkovic, Julia Kaltenborn, Yaniv Gurwicz, Peer Nowack, David Rolnick, Julien Boussard
arXiv Machine Learning
Sep 29

ALF: An Active Learning Framework for Scientific Discovery

ALF is a modular active learning framework designed to streamline the entire data acquisition process for scientific discovery. It offers a single API that supports both offline benchmarking against existing datasets and online deployment with an oracle for real‑world candidate acquisition. The framework is open‑source and available on GitHub.

By Shikha Surana, Alex Hawkins-Hooker, Olivia Gallup, Christoph Brunken, Jules Tilly, Paul Duckworth
arXiv Machine Learning
Sep 29

New LoRA Skills Should Read but Never Write

The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.

By Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
arXiv Machine Learning
Sep 29

Persistent Homology of Time Series through Complex Networks

The paper introduces a unified pipeline that classifies univariate time series by first converting them into graphs using one of five constructions from three families (visibility, transition, proximity). The resulting graph is turned into a dissimilarity matrix, from which a Vietoris–Rips filtration produces persistence diagrams that are vectorized via persistence landscapes and topological summary statistics. Experiments on twelve UCR benchmarks reveal that no single graph construction dominates, diffusion distance consistently outperforms shortest-path metrics, and persistence-based features remain robust to noise.

By \.Ismail G\"uzel