Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning

Sep 14

When Connected Does Not Mean Similar: Charting the Homophily Boundary of SNAP-KG for Streaming Entity Integration

SNAP‑KG is a framework that assigns new entities to semantic communities in a growing knowledge graph using only raw features, without graph access or retraining at inference time. The authors evaluate it on five multi‑view benchmarks and a large OGB‑WikiKG2 graph, all of which contain at least one homophilous view, and find strong performance. Extending the evaluation to three heterophilous graphs shows that when no homophilous view exists, clustering quality drops sharply for both SNAP‑KG and transductive baselines; the key factor is the homophily of the relation rather than the number of relations, and multi‑view fusion only helps if at least one homophilous relation is present. "whyItMatters":"The study reveals that SNAP‑KG’s effectiveness relies on the homophily assumption, highlighting a limitation for heterophilous knowledge graphs and suggesting a direction for future research on heterophily‑aware models."

By Jui-Chien Lin, Oshani Seneviratne

benchmarks

arXiv Machine Learning

Sep 14

Certified AI Triage of ICU Alarms

The paper introduces a certified AI triage system for ICU alarms, reframing alarm reduction as a three-way decision (retain, suppress, or defer). It demonstrates that, with a 5% budget, the system can suppress 74.8% of false ventricular‑tachycardia alarms while only silencing 1.5% of genuine ones, achieving an AUROC of 0.953 and a Challenge Score of 83.33—comparable to the best existing methods. The study also explores how grid granularity and calibration affect certification, showing that finer grids can certify fewer alarms but with tighter guarantees.

By Mohammed Sameer Syed, Rozhin Yasaei

benchmarks

arXiv Machine Learning

Sep 14

RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search

RiPPLE is a method for ranking neural architectures across an entire search space using only a small fraction of early training data. It treats partial training as labels for a limited set of anchor architectures, extrapolates their learning curves, and propagates these surrogate labels to other architectures without requiring per‑candidate features. The approach is evaluated on twelve benchmark cells from four search‑space families and the larger DARTS space, demonstrating its effectiveness in ranking quality, label efficiency, and architecture selection.

By Yifan Yang, Zhaoyan Wang, Zheng Gao, Xiaoyu Li, Jiaojiao Jiang

benchmarks

arXiv Machine Learning

Sep 14

MInTRL: Off-policy Intervention can boost On-policy RL

MInTRL (Minimal Intervention Reinforcement Learning) expands exploration in on-policy reinforcement learning by inserting sparse, local corrections into rollouts via a judge-intervention policy. These interventions replace erroneous suffixes and immediately return control to the main policy, allowing the agent to explore beyond its natural trajectory while maintaining on-policy data. The method uses a sequence-level advantage-regression objective, avoiding importance sampling, and demonstrates superior performance on math and code benchmarks compared to standard on-policy and off-policy baselines.

By Mingyu Chen, Yefan Tao, Gerald Friedland, Xuezhou Zhang, Chris Kong

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Sep 14

Correlation-Guided Fast Machine Unlearning via Hessian Analysis

The paper presents a fast machine unlearning method that uses Hessian analysis to identify correlated training data and applies a closed‑form update rule. This approach achieves an 82× speedup over traditional influence‑function unlearning while maintaining or slightly improving model accuracy. Experiments on seven dataset‑architecture pairs, including CIFAR‑100 with ResNet‑50, show strong forgetting performance and low vulnerability to membership inference attacks.

By Ayushi Thakur, Ruchir Gupta, Amit Kumar Jaiswal, Prayag Tiwari

benchmarks safety

arXiv Machine Learning

Sep 14

Write on Paper and Get the Online Digital Trace:\newline A New Era for Handwriting

The paper introduces a new method for capturing the digital trace of handwriting written on regular paper. It combines a sensor-equipped digital pen with advanced AI algorithms to reconstruct the pen’s trajectory in real time, eliminating the need for specialized styluses or substrates. The approach integrates hardware development and embedded AI software to enable seamless digitization of traditional handwriting.

By Florent Imbert, Yann Soullard, Eric Anquetil, Tanja Harbaum, Alexey Serdyuk, Fabian Kress, Tim Hamann, Peter Kampf

benchmarks

arXiv Machine Learning

Sep 14

SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics

SaFeR-Steer is a progressive multi‑turn alignment framework that uses staged synthetic bootstrapping and tutor‑in‑the‑loop GRPO to train a single student model against adaptive, on‑policy attacks. It introduces Trajectory‑Consistent Summative Reward (TCSR) to ensure that low‑quality turns impact the overall trajectory reward. The authors release the STEER dataset and demonstrate that SaFeR‑Steer significantly improves safety and helpfulness on both single‑turn and multi‑turn benchmarks for Qwen2.5‑VL models.

By Haolong Hu, Hanyu Li, Tiancheng He, Huahui Yi, An Zhang, Qiankun Li, Kun Wang, Yang Liu, Zhigang Zeng

reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Sep 14

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

The paper investigates nonlinear dimensionality reduction for Bayesian optimisation (BO) by transforming high‑dimensional black‑box optimisation problems into a sequence of low‑dimensional latent‑space BO (LSBO) tasks. It extends earlier linear embedding approaches by using variational autoencoders (VAEs), deep metric loss, and adaptive retraining to better capture nonlinear structure, and couples LSBO with sequential domain reduction (SDR‑LSBO) to progressively narrow search domains. Experiments on GPU‑accelerated BoTorch with Matérn‑5/2 Gaussian‑process surrogates show that VAE‑based LSBO outperforms adaptive linear embeddings, and the authors provide a theoretical analysis of latent‑space error versus representation gap under PAC‑Bayes conditions.

By Luo Long, Coralia Cartis, Paz Fink Shustin

rag diffusion benchmarks

arXiv Machine Learning

Sep 14

Limits of LLM Text Detectors in Education

The paper "Limits of LLM Text Detectors in Education" argues that existing LLM‑generated text detectors assume a binary human/LLM distinction, which fails to capture realistic student‑AI collaboration. It introduces a contribution‑aware evaluation framework with eight student contribution levels and presents GEDE, a benchmark of over 900 human‑written and 12,500 generated essays across 886 tasks. Using GEDE, the authors evaluate four detection methods and find that most detectors perform poorly on intermediate contribution levels, especially LLM‑assisted revisions, raising concerns about false accusations.

By Lukas Gehring, Benjamin Paa{\ss}en

llms benchmarks