Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision
Sep 22

R3D: Revisiting 3D Policy Learning

arXiv:2604.15281v2 Announce Type: replace Abstract: 3D policy learning promises superior generalization and cross-embodiment transfer, but progress has been hindered by training instabilities and sev...

By Zhengdong Hong, Shenrui Wu, Haozhe Cui, Boyi Zhao, Ran Ji, Yiyang He, Hangxing Zhang, Zundong Ke, Jun Wang, Guofeng Zhang, Jiayuan Gu
arXiv Machine Learning
Sep 22

Role-Aware Morgan Fingerprints for Reaction Yield Prediction

The paper introduces MFP, a reaction yield prediction method that uses role-aware Morgan fingerprints. It computes count-based circular fingerprints for each reaction component, aggregates them by chemical role, and combines them with transformation-sensitive difference features into a fixed-length descriptor for a feed-forward neural regressor. On the Suzuki‑Miyaura and Buchwald‑Hartwig benchmarks, MFP achieves R² scores of 0.878 and 0.969 respectively, while training an order of magnitude faster than graph or Transformer-based alternatives.

By Chinmay Mirji, Prashant Shekhar, Foram Madiyar, Hao Peng
arXiv Machine Learning
Sep 22

UniGIO: Unified Generative Global In-situ Weather Modeling from Spatiotemporal Incomplete Observations

UniGIO is a generative framework that models global in‑situ weather dynamics directly from incomplete GIO data, unifying forecasting, imputation, and generation across arbitrary missing ratios. It employs an Observation Mixer, Event Aligner, Adaptive Temporal Mixer, and a Mixture‑of‑Experts structure to capture station‑level complementarity, temporal dependencies, and extreme events, refining outputs with a Local Refiner. Experiments on the Weather‑5K dataset show state‑of‑the‑art performance, improving accuracy, fidelity, and extreme event capture by 11%, 12%, and 5% respectively.

By Songru Yang, Zili Liu, Tao Han, Ben Fei, Lei Bai, Chang Liu, Zhengxia Zou, Xiangyang Ji, Wanli Ouyang, Zhenwei Shi
arXiv Machine Learning
Sep 22

Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles

The paper introduces mutation analysis as a metric for evaluating GPU‑kernel benchmark oracles, injecting over ten thousand faults into verified CUDA implementations of 188 KernelBench problems. It shows that the current official checkers miss 16.9% of faults, with precision faults being especially problematic, and demonstrates that optimized test suites can achieve 98% detection with only two inputs per problem. The study also reveals flaws in existing patches and a fuzzing recipe that incorrectly rejects correct kernels 107 times.

By Mingzhe Du, Anh Tuan Luu, Dong Huang, See-Kiong Ng
arXiv Machine Learning
Sep 22

Benchmarking Hybrid Deep Learning Architectures for Predictive Maintenance in Industry 4.0

The study evaluates six deep learning architectures for predictive maintenance in Industry 4.0, focusing on Recurrent Neural Networks (RNNs) and Transformers. It finds that Transformers perform well on stable, slow-moving data but overreact to noisy, chaotic data, whereas a hybrid model combining an LSTM layer with a Transformer layer better filters noise and delivers more consistent predictions. The hybrid approach improves accuracy and reliability across varying levels of data volatility.

By Zhengyang (Cissy), Gu, Joseph E. Hernandez, Thomas Cook, John Burtenshaw, Sean Scott, Chris Couch
arXiv Machine Learning
Sep 22

Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting

The paper introduces paired, mechanism‑controlled stress tests that decompose changes in expected squared error for time‑series forecasting into environmental risk and forecast‑oracle distance. Using an origin‑conditioned predictive oracle, the authors validate three end‑to‑end controls and apply the benchmark to 24 forecasters, revealing that many models exhibit higher realized MSE yet lower oracle distance under frequent switching, and that environmental risk dominates in most scenarios. The study also demonstrates that visually compelling discovery profiles often fail to replicate on independent data‑generating process realizations, underscoring the importance of component‑wise diagnosis and held‑out stability audits.

By Xu Lin (Tsinghua University, Beijing, China), Runheng Zuo (Tsinghua University, Beijing, China), Shengxuan Xu (Tsinghua University, Beijing, China), Qitai Tan (Tsinghua University, Beijing, China), Hongyu Lin (Tsinghua University, Beijing, China), Xiao-Ping Zhang (Tsinghua University, Beijing, China)
arXiv Machine Learning
Sep 22

When Does Adversarial Refinement Help? A Negative Result and Open Problem in Adapting R3GAN to Time Series Imputation

The paper investigates whether the stable GAN architecture R3GAN can improve time‑series imputation when adapted to 1‑D temporal data. Using a coarse‑to‑fine refinement framework and a frequency‑domain discriminator, the authors evaluate 14 saved configurations across three datasets and find a negative result: most configurations either show negligible improvement or degrade performance compared to baseline methods. The study highlights that the usual argument—GANs optimize distributional objectives rather than point‑wise ones—does not fully explain the lack of benefit, and it poses an open problem regarding why a learned discriminator fails to provide useful refinement gradients while diffusion denoisers succeed, offering practical guidance on when adversarial refinement may be worthwhile.

By Yufeng He
arXiv Machine Learning
Sep 22

Tool-Augmented On-Policy Distillation for LLM Domain Adaptation in Sequence-Based Omics Tasks

The paper introduces OmicsBench, a new reasoning benchmark for multi‑omics sequences that includes 1,160 expert‑validated questions across DNA regulation, RNA processing, and protein function tasks, requiring traceable evidence chains. Evaluation of 17 large language models shows that scientific LLMs, while more accurate in classification, often lack valid evidence, suggesting shortcut learning. To address this, the authors propose tool‑augmented on‑policy distillation (TA‑OPD), a post‑training method that improves both evidence grounding and predictive performance across five Qwen3.5 models of varying sizes.

By Jie Ying, Zhefan Wang, Zihong Chen, Zhengqing Li, Jinzhe Li, Gang Li, Jian Liu, Fang Hu, Tao Luo, Zhonghang Yuan, Wanli Ouyang, Stan Z. Li, Fan Yang, Nanqing Dong