Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning
Sep 24

When Labels Are Scarce: An Oscillatory State Space Model for Vibration Diagnosis

The paper introduces DualRes, a compact oscillatory state‑space model designed for vibration‑based fault diagnosis when labeled data are scarce and computational resources are limited. DualRes integrates two spectral views of vibration and employs selective oscillatory memory to learn how long to retain temporal patterns, resulting in a lightweight encoder with only 39,528 parameters. Evaluations on six bearing datasets and a gearbox benchmark show that DualRes outperforms nine competing methods across most label budgets, achieving significant gains in macro‑F1, faster inference, and reduced storage requirements.

By Mainak Mallick, Seung-Kyum Choi
arXiv Machine Learning
Sep 24

Learning Where to Look: A Shared Relative-Alignment Module for Time-Series Forecasting and PPG-to-Vital-Sign Reconstruction

The paper introduces ROOSTER, a shared relative‑alignment module that learns how to map a condition sequence to a target sequence for both PPG‑to‑vital‑sign reconstruction and long‑horizon multivariate time‑series forecasting. ROOSTER uses a learnable periodic‑comb bias over the target‑condition offset, allowing it to discover identity alignment or seasonal lags and report the chosen correspondence. Experiments show that ROOSTER outperforms existing baselines on heart‑rate and respiratory‑rate reconstruction from wrist‑worn photoplethysmograms, and achieves the lowest horizon‑averaged MSE on four forecasting benchmarks, outperforming the underlying forecasting model in most dataset‑horizon settings.

By Ragamayi Puli, Shunya Nagashima
arXiv Machine Learning
Sep 24

VCMM: Variance-Calibrated Momentum for Multimodal Learning

VCMM: Variance-Calibrated Momentum for Multimodal Learning proposes a new optimizer that adapts momentum based on modality-specific gradient dynamics. It estimates minibatch noise and temporal drift online, using a Kalman-inspired controller to set modality-specific momentum and applies bias correction for the first moment. Experiments on four multimodal benchmarks show consistent improvements with modest training overhead.

By Zhongjing Gu, Chenyang Huang, Yufa Feng, Chong He, Qinxu Ding, Yiming Cui
arXiv Machine Learning
Sep 24

MENO: Memory-Efficient Neural Operator

The paper introduces MENO, a Memory‑Efficient Neural Operator designed for solving partial differential equations (PDEs). MENO leverages a Manifold Function Encoder to achieve a small memory footprint that does not depend on data resolution, enabling faster training and potential scalability to large models. It accepts PDE inputs of arbitrary form—including different geometric domains and discretizations—allowing cross‑geometry scenarios, and demonstrates strong generalization with superior accuracy on most tested benchmarks.

By Shengyang Xu, Weijun Zhang, Jun Hu, Pengzhan Jin
arXiv Machine Learning
Sep 24

Pheno-GS: Phenoscape-scale Geodesic Sinkhorn

Pheno-GS is a new method for computing scalable, geometry-aware optimal transport distances between large patient cohorts of single‑cell data. It uses graph connectivity regularization, an unbalanced OT formulation with KL marginal penalties, and a batched matrix algorithm that dramatically speeds up pairwise distance calculations. The authors validate the approach on synthetic benchmarks and a CyTOF perturbation dataset.

By Alistair Wilkinson, Christopher J. Tape, Smita Krishnaswamy
arXiv Machine Learning
Sep 24

Quantifying the Occult: A Comparative Study of Hindu and Buddhist Deities Using Machine Learning Methods

The paper presents a dual‑matrix computational framework that quantifies morphological and theological differences among 196 Hindu and Vajrayana Buddhist esoteric deities. It uses a Gower distance matrix with a new Cardinality Weighting algorithm for physical form and dense vector embeddings from LLMs for theological function, revealing how visual forms can mask shared functions and how high‑cardinality symbols cluster orthodox and Tantric entities. The study demonstrates near‑identical coordinates for the Hindu Chinnamasta and Buddhist Chinnamunda, and releases the architecture as an open‑source tool for Digital Humanities research.

By Ankit Bhattacharjee
arXiv Machine Learning
Sep 24

Giving Credit Where It's Due: Redundancy-Aware Learning for Efficient Reasoning

The paper introduces RECAP, a redundancy-aware credit assignment method that improves reasoning efficiency in large language models by assigning credit to each reasoning step based on its downstream role and contribution to the correct answer. RECAP uses a semantic dependency graph to measure structural responsibility and evaluates step efficacy via changes in gold-answer log-likelihood, enabling step-specific updates without requiring a separate reward model or concise trajectories. Experiments on two 7B models across four mathematical reasoning benchmarks show that RECAP enhances the accuracy-efficiency trade-off, boosting pass@1 by 2.0–3.7 percentage points while cutting reasoning tokens by 8–31% compared to GRPO.

By Yuqing Zhou, Hong Wang, Manqing Mao, Zhuoer Wang, Samson Koelle, Jie Yuan, Yanjun Lin, James Feng, Nikki Lijing Kuang, Ziwei Zhu, Wei Niu
arXiv Computer Vision
Sep 24

Tackling fluffy clouds: robust agricultural field boundary delineation from Sentinel-1 and Sentinel-2 satellite image time series

arXiv:2409.13568v3 Announce Type: replace Abstract: Accurate delineation of agricultural field boundaries is essential for effective crop monitoring and resource management. However, competing method...

By Foivos I. Diakogiannis, Zheng-Shu Zhou, Jeff Wang, Gonzalo Mata, Dave Henry, Roger Lawes, Amy Parker, Peter Caccetta, Suzanne Furby, Rodrigo Ibata, Ondrej Hlinka, Jonathan Richetti, Kathryn Batchelor, Chris Herrmann, Andrew Toovey, John Taylor
arXiv AI
Sep 24

Math Reasoning in LLMs is Organized by Approach, Not Topic

The paper argues that large language models (LLMs) organize their internal mathematical reasoning by reusable reasoning approaches rather than by the benchmark topics they are tested on. Using a generation‑replay protocol, the authors extract activation‑importance signatures from eight models across five math sources, cluster these signatures, and find that the resulting groups align more closely with reasoning approaches than with topics. The study shows that changing the requested reasoning approach shifts cluster assignments, while paraphrasing the prompt does not, underscoring the primacy of approach over topic in LLM reasoning.

By Sajad Goudarzi, Samaneh Zamanifard, Moloud Nasiri, Hamed Rahimian
arXiv AI
Sep 24

Distillation for Efficient Multitask Manipulation Policies via Conditional Flow Matching

The paper proposes a method to train efficient multi‑task manipulation policies by distilling knowledge from single‑task Conditional Flow Matching (CFM) experts. Instead of training separate models for each task, the authors transfer the experts’ learned velocity fields into a shared policy, combining this distillation signal with the original CFM objective. Experiments on RLBench demonstrate that this approach improves multi‑task performance while keeping the model size fixed, avoiding the need for larger capacity or performance drops seen with naive concatenated training.

By Shreya Deshmukh, Imen Mahdi, Nick Heppert, Abhinav Valada