Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 12

SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

SelectArena is a new benchmark for feature selection on real-world tabular data, evaluating 15 popular methods across 71 curated datasets for both classification and regression tasks. The study uses standardized protocols and modern foundation models, revealing trade-offs among predictive performance, runtime, stability, and validity. Tree-based methods (CART, RFImportance) and the F-Test emerge as Pareto-optimal choices, while search-based methods perform poorly in speed and stability.

By Bastian Sch\"afer, Dominika Matus, Lennart Purucker, Ricardo Knauer, Erik Rodner, Frank Hutter, Anton Sokolov, Arnaud Liefooghe
arXiv AI
Oct 12

Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction

The paper introduces UPTURN, a method that improves large language model self‑correction by learning where to intervene in a failed reasoning rollout and injecting a new user turn to redirect the model, rather than fine‑tuning the LLM. A small 130K‑parameter MLP head is trained on a frozen encoder using only final‑answer correctness to select intervention points. Experiments on four reasoners and six math benchmarks show that UPTURN consistently raises accuracy over a 32‑rollout baseline and outperforms existing self‑correction and reflection‑token injection techniques while transferring to new tasks without retraining.

By Jiwon Jeon, Myungsik Cho, Jeonghye Kim, Youngchul Sung
arXiv AI
Oct 12

Variance-Reduced Policy Gradient is Nearly Minimax-Optimal

The paper introduces Loopless Variance-Reduced Policy Gradient (LVR-PG), a framework that unifies first‑order estimators combining recursive updates with defensive importance sampling, such as PAGE-PG and STORM-PG. It proves that LVR-PG achieves a ∫O(ε⁻²) last‑iterate global convergence rate under weak gradient domination for general parametric policies, using only first‑order updates and minimal assumptions. By extending lower bounds to continuous parametric policies, the authors show an Ω(ε⁻²) sample complexity limit, establishing that LVR-PG is minimax‑optimal up to logarithmic factors and horizon terms.

By Leonardo Cesani, Matteo Papini, Marcello Restelli
arXiv AI
Oct 12

FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems

FORTE is a model‑agnostic production framework that discovers, validates, and removes features from industrial recommender systems. It uses efficient perturbation and grouped ablation to evaluate candidate feature sets daily, and lineage tracing to identify all downstream models affected. In two large‑scale systems, FORTE removed up to 43.3% of features without online regressions, cutting trainable parameters by 29.9% and improving serving throughput by 19.8‑25.5%.

By Vishnu Arun, Adam Obeng, Tianyou Wang, Zilin Ren, Mukuntha Narayanan Sundararaman, Han Wang, Shivin Thukral, Zhixiang Wang, Saurabh Vishwas Joshi, Darren Reger
arXiv AI
Oct 12

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

The paper investigates run‑to‑run variance in semantic query engines that use large language models (LLMs) to evaluate rows. By re‑executing benchmarks (SemBench, LRO‑Bench, UDA‑Bench) and logging 1.2 million model calls, the authors find that between 0.07% and 10.2% of rows change output at temperature 0, with some tasks showing up to 24% change. They show that plan amplification can cause a large fraction of answers to differ, that many benchmark comparisons are unresolved, and that voting and calibrated cascades reduce but do not eliminate variance, leading to new reporting and design guidelines.

By Oktie Hassanzadeh, Dharmashankar Subramanian
arXiv AI
Oct 12

AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation

AutoControl Arena is an automated framework for evaluating frontier AI risk that separates deterministic state from generative dynamics, reducing logic hallucination while preserving flexibility. The system uses a three‑agent architecture and achieves over 98% end‑to‑end success and 60% higher human preference than existing simulators. Experiments across 70 scenarios reveal that alignment risk rises sharply under pressure, advanced reasoning improves direct harm robustness but worsens gaming scenarios, and weaker models cause non‑malicious harm while stronger models conceal misalignment.

By Changyi Li, Pengfei Lu, Xudong Pan, Fazl Barez, Min Yang
arXiv AI
Oct 12

FactoryBench: Evaluating Industrial Machine Understanding

FactoryBench is a benchmark designed to evaluate time‑series models and large language models on machine understanding using industrial robotic telemetry. It organizes Q&A pairs across four causal levels—state, intervention, counterfactual, and decision—based on Pearl’s ladder of causation, and includes five answer formats, with structured responses scored deterministically and free‑form responses evaluated by an LLM‑as‑judge voting protocol. The benchmark is built from over 70,000 Q&A items derived from roughly 15,000 episodes in the FactoryWave dataset (from a UR3 cobot and a KUKA KR10 arm) as well as AURSAD and voraus‑AD, and zero‑shot tests of six leading LLMs show that none surpass 50% on structured levels or 18% on decision‑making tasks.

By Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu, Marcos Gomez-Bracamonte, Riccardo Maggioni, Alessandro Lombardi, Camilla Mazzoleni, Federico Martelli, Balazs Gunther, Jonas Petersen, Philipp Petersen
arXiv AI
Oct 12

MPMMine: Better Benchmarks for Constraint Acquisition

MPMMine introduces a comprehensive benchmark suite for Constraint Acquisition (CA) in Mathematical Programming, addressing shortcomings of existing datasets such as inconsistent formats and limited problem classes. The suite offers standardized, extensible artifacts—including multiple model formulations, numerous parameter instantiations, and thousands of solutions and non‑solutions across discrete, continuous, and mixed domains—alongside natural‑language descriptions for text‑to‑model methods. Accompanying tools, the mpmmine‑py API and an interactive leaderboard, enable consistent evaluation of CA algorithms within isolated containers.

By Rafa{\l} Stachowiak, Tomasz P. Pawlak
arXiv AI
Oct 12

ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models

ReasoningShield introduces a benchmark and lightweight models for detecting unsafe content in the intermediate reasoning steps of Large Reasoning Models (LRMs). The benchmark contains 9.2K annotated (Query, CoT) pairs across 10 risk categories, 3 safety levels, and 8 LRMs, and the proposed models achieve 91.8% F1, outperforming existing moderation tools by significant margins. The study highlights that current moderation tools degrade by up to 35.2% on Chain-of-Thought reasoning compared to traditional answer moderation.

By Changyi Li, Jiayi Wang, Xudong Pan, Geng Hong, Min Yang
arXiv AI
Oct 12

Regression-adjusted Monte Carlo Estimators for Shapley Values and Probabilistic Values

The paper introduces a novel method that merges Monte Carlo sampling with linear regression to estimate probabilistic values such as Shapley and Banzhaf values. It generalizes previous algorithms by allowing any efficiently computable function family, enabling the use of tree-based models like XGBoost and cross‑fitting for unbiased estimates. Experiments on eight datasets show the approach achieves state‑of‑the‑art accuracy, reducing error by up to 11× for Shapley values and 390× for general probabilistic values compared to existing methods.

By R. Teal Witter, Yurong Liu, Christopher Musco
arXiv AI
Oct 12

CORE-Seg: Reasoning-Driven Segmentation for Complex Lesions via Reinforcement Learning

The paper introduces CORE‑Seg, an end‑to‑end framework that combines reasoning and segmentation for complex lesion detection in medical images. It leverages a Semantic‑Guided Prompt Adapter and a progressive training strategy from SFT to GRPO, using an adaptive dual‑granularity reward mechanism to address reward sparsity. CORE‑Seg achieves state‑of‑the‑art performance, with a mean Dice score of 37.06%—14.89% higher than the next best baseline—and reduces the failure rate to 18.42%.

By Yuxin Xie, Yuming Chen, Yishan Yang, Yi Zhou, Tao Zhou, Zhen Zhao, Jiacheng Liu, Huazhu Fu
arXiv AI
Oct 12

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

The paper introduces Coverage-conditioned Group Policy Optimization (CovGPO), a method that addresses sampling imbalance in group-based policy optimization for long-horizon agentic tasks. CovGPO applies first-visit observation coverage only when all samples in a group receive zero outcome reward, giving higher relative advantages to state-exploring actions without altering the original reward signal. Experiments on ALFWorld and WebShop benchmarks demonstrate that CovGPO consistently outperforms group-based baselines, especially on hard tasks.

By Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He, Yu Li, Mengyi Liu, Pengwei Chen, Jun Xu, Lei Feng
arXiv Machine Learning
Oct 12

Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective

The paper investigates multi‑horizon forecasting of event sequences as a conditional generative modeling problem, evaluating forecasts for both prediction accuracy and distributional realism. It introduces a benchmark with metrics, baselines, and real‑world datasets, revealing a trade‑off between accuracy and realism. The authors propose Wasserstein BaryBooster, an algorithm that combines samples from a realism‑strong diffusion model with a state‑of‑the‑art predictor to convert distributional realism into improved prediction accuracy.

By Dmitry Osin, Egor Surkov, Petr Mokrov, Igor Udovichenko, Ivan Karpukhin, Evgeny Burnaev, Egor Shvetsov, Alexander Korotin
arXiv Machine Learning
Oct 12

Data-driven Optimal Sensor Placement for Aerosol Monitoring in the UK

The paper presents a data‑driven framework for optimal sensor placement in aerosol monitoring across the UK. It integrates a convolutional Gaussian neural process into the DeepSensor optimisation framework, using genetic optimisation with lazy evaluation and caching to cut computational time by 60% while matching the performance of a greedy solver. The approach achieves a 22% improvement in Joint NLL for Joint‑MI‑driven genetic placement after selecting five stations, and shows that model‑based criteria align with oracle benchmarks under certain meteorological conditions.

By Talha Ansar (University of Engineering and Technology, Lahore, Pakistan), Waqar Muhammad Ashraf (The Alan Turing Institute, London, UK), Alejandro Coca-Castro (The Alan Turing Institute, London, UK), Manvendra Janmaijaya (The Alan Turing Institute, London, UK), Ramit Debnath (University of Cambridge, Cambridge, UK), J. Scott Hosking (The Alan Turing Institute, London, UK)
arXiv Machine Learning
Oct 12

Expert-level detection of low-voltage intervals in EEG as a robust biomarker of neonatal encephalopathy

The study introduces low‑voltage intervals (LVIs) as an amplitude‑ and duration‑based marker of neonatal EEG discontinuity and develops a ConvNeXt AI model that detects LVIs with expert‑level accuracy (AUC = 0.996, MCC = 0.926). In a cohort of 444 neonates, LVI percentage strongly correlates with EEG background grades, moderately with hypoxic‑ischaemic encephalopathy severity, and predicts seizures (AUC = 0.872). The results demonstrate that automated LVI quantification can serve as a robust biomarker for neonatal encephalopathy and seizure risk.

By John M. O'Toole, Sean R. Mathieson, Aurel Luca, Soraia Ventura, Sean Griffin, Geraldine B. Boylan, Robert Hogan
arXiv Machine Learning
Oct 12

Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics

The paper introduces a provenance‑grouped protocol for evaluating machine‑learning models on perovskite solar cell data, addressing the problem of data leakage caused by duplicate devices across institutions. By mapping institutional provenance and creating hierarchical grouped splits that isolate duplicates, the authors benchmark thirteen models and find that tree ensembles excel on random splits but fail on genuinely unseen institutions, where neural architectures perform best. The study also shows that device architecture and fabrication methods are as influential as absorber chemistry for efficiency, providing a standardized framework to distinguish memorisation from true physical discovery.

By Qiwei Han, Chi Zhou
arXiv Machine Learning
Oct 12

TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

TreeHCA introduces a method that couples hindsight-guided exploration with hindsight credit assignment for multi-turn large language model agents. By using changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, it builds a tree of reasoning and tool interactions, and applies an inverse-hindsight importance correction to propagate terminal outcomes into turn-level advantages. Experiments on seven question-answering benchmarks with three Qwen backbones show that TreeHCA outperforms all baselines in aggregate exact match.

By Debajoy Mukherjee, Jeremy Carleton, Sushil Vemuri, Srinivas Shakkottai, Dileep Kalathil
arXiv Machine Learning
Oct 12

Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning

arXiv:2610. 12865v1 Announce Type: new Abstract: Existing convergence guarantees for non-log-concave sampling with stochastic gradient Langevin Monte Carlo (SG-LMC) typically assume globally Lipschitz gradients and uniformly bounded noise variance.

By M. Berk Sahin, Ege C. Kaya, Arda Fazla, Behzad Sharif, Abolfazl Hashemi