Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 25

DeGRe: Dense-supervised Generative Reranking for Recommendation

DeGRe is a dense‑supervised generative reranking framework designed to improve multi‑stage recommender systems by addressing label bias and credit assignment issues. It uses an offline Lookahead Evaluator with beam search to generate dense supervision signals, which are distilled into a lightweight Online Generator that can perform efficient greedy decoding at inference time. Experiments show that DeGRe outperforms baselines on public benchmarks and industrial datasets, and it has been successfully deployed on Taobao Flash Shopping to enhance online recommendations.

By Chaotian Song, Jingyao Zhang, Chenghao Chen, Zisen Sang, Dehai Zhao, Guodong Cao, Boxi Wu, Deng Cai, Jia Jia
arXiv Machine Learning
Sep 25

Uncovering Residential PV-EV Co-Adoption from Smart-Meter Data: Load Archetypes and Detection for Demand-Side Planning

The paper presents a two-part workflow for analyzing smart‑meter data to uncover patterns of residential co‑adoption of photovoltaic (PV) systems and electric vehicles (EVs). First, dynamic time warping k‑means clustering identifies distinct daily import/export archetypes for PV‑only, EV‑only, co‑adopters, and neither groups, revealing a midday‑centered export pattern for many co‑adopters. Second, a bidirectional LSTM model trained on 21‑day windows achieves high detection performance (AUROC 0.991, macro‑F1 0.906) for PV/EV activity, outperforming tabular baselines and remaining robust across labeling rules and temporal splits.

By Jack Zheng, Hao Wang
arXiv Machine Learning
Sep 25

OPDiv: Optimal Selection of Top-K High-Scoring, Diverse Compounds

OPDiv is a new algorithm that addresses the tradeoff between ranking quality and chemical diversity in virtual screening. It uses integer optimization to select an optimal subset of top‑k compounds that meet a specified diversity threshold, evaluated with fingerprint, shape, and electrostatic metrics. The method demonstrates how to benchmark virtual screening pipelines by comparing their best achievable diverse selections.

By Miroslav L\v{z}i\v{c}a\v{r} (Deep MedChem)
arXiv AI
Sep 25

Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

The paper presents Baszta, a Polish multi‑label content‑safety classifier trained by fine‑tuning the 124M‑parameter allegro/herbert‑base‑cased model on five categories (hate, vulgarity, sexual content, crime, self‑harm) using a Focal + R‑Drop objective. In out‑of‑distribution evaluation on the Gadzi Język benchmark, Baszta achieves a small but statistically significant improvement in micro‑F1 over the Bielik Guard system, though the macro‑F1 advantage disappears when both models are properly tuned. The study also explores calibration techniques, showing that per‑category temperature scaling can recover performance lost by Platt scaling or isotonic regression, and discusses the trade‑offs between robust calibration and adversarial recall.

By Adam G\'orski, Mateusz J\k{a}kalak, Rafa{\l} Jakubowski
arXiv AI
Sep 25

Breaking the Environment Wall: Evolving LLM Agent Environments for Recursive Self-Improvement

The paper introduces Env‑Rethink, a 27B post‑trained model system designed to help large language model agents better interact with complex, evolving environments. It builds Collection Maps and Event Logs to organize scattered information, uses offline trajectory learning to detect noise, and generates virtual event histories to evolve environments for more challenging tasks. Experiments show that Env‑Rethink improves downstream task performance by over 15.1% rubric pass rate across nine models on 30 tasks.

By Yukai Wu, Yuanjing Yang, Le Zhou, Shaokun Han, Haoyu Wang, Zirui Tang, Weihuang Zheng, Maxm Pan, Xuanhe Zhou, Fan Wu
arXiv Machine Learning
Sep 25

BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularization

BranchShine-CR is a 25‑million‑parameter model that transcribes multilingual speech into the International Phonetic Alphabet (IPA). It uses log‑mel features, a rotary‑position E‑Branchformer encoder, intermediate self‑conditioned CTC, and consistency regularization across augmented views. On 16,646 shared IPApack++ test utterances it achieves a 4.47 % IPA character error rate, a 22.3 % relative improvement over ZIPA‑CTC‑NS and outperforms a similarly sized NeMo Conformer baseline across all 41 language labels.

By Nikhil Navas, Sergio Chevtchenko, Talisson Damiao, Saeed Afshar
arXiv Machine Learning
Sep 25

Neural Transport Nested Sampling

Neural Transport Nested Sampling (NTNS) is a new sampling algorithm that merges nested sampling with neural flow-based methods to sample from Boltzmann distributions of molecular systems. It employs a flow-matching velocity as the drift in a Metropolis–Hastings corrected Langevin kernel within a nested sampling loop, requiring only target energy evaluations and enabling scalable estimation of the full partition function for high-dimensional particle systems. Benchmarks on Lennard–Jones clusters up to 55 particles show NTNS reduces interatomic distance and energy Wasserstein errors by more than an order of magnitude compared to leading neural baselines, while also providing a calibrated, temperature-resolved partition function estimate that captures phase structure from a single run.

By David Yallup, Will Handley
arXiv Machine Learning
Sep 25

Lightweight Probabilistic Downscaling from a Deterministic Base Model

The paper introduces lightweight probabilistic downscaling models that build on a modified U‑Net backbone, adapting two recent machine learning techniques from weather forecasting. Using a two‑stage training curriculum—deterministic pretraining followed by probabilistic fine‑tuning—the authors evaluate their models on the CORDEX‑ML‑Bench suite for daily maximum temperature and precipitation in the Alps, New Zealand, and South Africa. The results show that this approach outperforms the current state‑of‑the‑art in RMSE, offering a more computationally efficient method for generating fine‑resolution regional climate data.

By Joseph McLean, Tiffany Vlaar, Sigrid Passano Hellan, Linus Ericsson
arXiv Machine Learning
Sep 25

When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection

The paper investigates how repeated rows in released datasets—often treated as i.i.d. samples—introduce a hidden measurement layer that affects anomaly detection. It shows that identical rows can cap evaluation performance, make AUROC sensitive to replication, and bias detectors toward multiplicity size. The authors audit 690 OddBench datasets, find significant train-test overlap and label conflicts, and propose SCOUT, a support‑count orthogonalized detector that separates replication‑invariant evidence from exposure‑aware counts, achieving comparable or better AUROC while controlling false‑positive rates.

By Jie Deng
arXiv AI
Sep 25

Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

Synthetic Hospital is an open, fully synthetic longitudinal electronic health record benchmark created from public medical education material. It contains 1,268 patients and 5,602 encounters, with every diagnosis, finding, and temporal relation grounded in standard ontologies and traceable back to its source. Physicians could not reliably distinguish its records from real charts, and current frontier AI models perform significantly below human experts on tasks such as reconstructing problem lists and summarizing charts.

By Christine Park, Valerie Chen, Tim Dettmers
arXiv Machine Learning
Sep 25

GBFRVFL: Granular-Ball Computing-Based Fuzzy Random Vector Functional Link Network

The paper introduces GBFRVFL, a fuzzy granular-ball random vector functional link network designed to improve robustness in noisy, imbalanced, or uncertain data settings. It employs granular-ball computing to group raw samples into adaptive balls and proposes two membership assignment schemes: F-GBRVFL, which uses fuzzy membership to gauge ball reliability, and SDAP-GBRVFL, which introduces a statistical density‑adaptive Pythagorean membership that adjusts based on class variance, local sparsity, and ball compactness. Experiments on 37 UCI and KEEL datasets show that these models outperform baseline methods in both clean and noisy conditions, achieving higher accuracy and stability.

By A. Quadir, A. Rahaman, P. N. Suganthan, M. Tanveer
arXiv Machine Learning
Sep 25

SwitchPFN: Shared Switching Dynamics for Frozen In-Context Time Series Classification

SwitchPFN introduces a shared projection and regime codebook for time‑series classification with tabular foundation models, preserving local temporal transitions while ensuring consistent feature definitions across samples. The method outperforms existing representations, achieving the highest mean accuracy on evaluated benchmarks and improving the strongest baseline by 4.47% relative. Ablation, sensitivity, and limited‑data experiments confirm the benefits of the proposed representation design.

By Zhenyi Zhu, Jacqueline Pang, Peilin Shen, Tianyi Song, Tingwei Zhang, Keyi Hu, Kangjun Yin, Shiwei Pu, Yingbo Zhou, Chen Shao
arXiv Machine Learning
Sep 25

MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization

MF-SCBO is a new multi‑fidelity extension of Scalable Constrained Bayesian Optimization designed for high‑dimensional black‑box functions with black‑box constraints. It handles an arbitrary number of fidelity levels and non‑nested sampling, addressing gaps in existing methods. Experiments on standard benchmarks and challenging problems show that MF‑SCBO generally converges faster than both single‑fidelity SCBO and other multi‑fidelity approaches in high‑dimensional constrained settings.

By Lucas Palazzolo, Micka\"el Binois, La\"etitia Giraldi
arXiv Machine Learning
Sep 25

Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility

The paper introduces FROST, an online framework that filters synthetic training data by estimating its utility through gradient feedback anchored in real data. FROST calibrates batch utility against recent history to decide when to filter, removing 20–30% of synthetic samples while improving performance on image classification and LLM fine-tuning tasks. The method is also applied to a large‑scale industrial ads re‑ranking system, yielding significant gains over an optimized production baseline.

By Yanran Wu, Sana Lakdawala, Renzo Tassara Miller, Chongyang Bai, Sharath Ciddu, Shivendra Pratap Singh, Kungang Li, Sandeep Pandey, Chunwei Liu
arXiv Machine Learning
Sep 25

Reconstructing short-lived particles using hypergraph representation learning

The paper introduces HyPER, a hypergraph-based graph neural network architecture designed to reconstruct short-lived particles in collider experiments. By leveraging hypergraph representation learning, HyPER builds more powerful and efficient representations of collider events, enabling accurate reconstruction of parent particles from final-state objects. In simulations, HyPER outperforms existing state‑of‑the‑art techniques while using fewer parameters, and its flexible hypergraph approach can be applied to a wide range of physics processes.

By Callum Birch-Sykes, Brian Le, Yvonne Peters, Ethan Simpson, Zihan Zhang
arXiv Machine Learning
Sep 25

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

The paper investigates whether machine learning models for IoT intrusion detection truly learn attack patterns or rely on dataset shortcuts. Using the CyberFlowIoT-GICAP benchmark, the authors evaluate four learning paradigms across different feature sets and split strategies, finding that performance is largely driven by feature representation and that tree-based models can exploit temporal artifacts. The study also highlights asymmetric attack detectability and proposes a four-point protocol checklist for realistic evaluation.

By Uday Shankar Roy, Mahbuba Jahan Minu
arXiv Machine Learning
Sep 25

Same Bit Width, Different Outcomes: Post-Training Quantization of Text-to-Speech Across Architectures

The paper evaluates post‑training quantization (PTQ) for text‑to‑speech (TTS) models across multiple architectures using a unified protocol. It shows that reducing weights to 4‑bit per‑channel can significantly lower predicted mean opinion scores (UTMOS) and that even 8‑bit per‑tensor scaling can cause severe degradation, with the impact varying by model. A staged ablation identifies the sensitive components, and per‑layer GPTQ can recover performance to within 0.1 UTMOS, while real int8 and int4 kernels confirm the simulated results on hardware, demonstrating that each configuration must be validated on the target runtime.

By Se Un Park, Yutae Kim, Junyoung Park
arXiv Machine Learning
Sep 25

Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

The study investigates whether large language models (LLMs) can automatically improve the harnesses used for hardware design verification. By evolving harnesses around a fixed subject model on 12 proprietary root‑cause localization tasks, the researchers found that automatically evolved harnesses increased completed attempts by 71‑76% and task coverage by 80‑100%, though overall correct attempts improved only 18‑24%. Despite these gains, the evolved harnesses did not consistently consolidate into a single dominant solution across tasks and metrics, and a separate cross‑benchmark case showed that a repair harness could yield a 35.6% increase in functional passes over a baseline.

By Kidus Seyoum, Ajay Mittur
arXiv Machine Learning
Sep 25

Where Hallucinations Live: A Cross-Architecture Circuit in VQ-Tokenized Vision-Language Models

The paper investigates why vision‑language models that tokenize images with vector‑quantized (VQ) codebooks frequently hallucinate objects on grounded yes/no tasks. By applying activation patching across 25 models from eight large‑language‑model families, the authors uncover an early‑layer attention routing circuit shared by VQ‑tokenized VLMs. They develop a three‑gate diagnostic that isolates ten models carrying this circuit, show that swapping a single architectural component (VQ+Linear) introduces the circuit, and demonstrate that ablating the early‑layer ($L_0$) component reduces hallucinations in open‑ended generation by 31 % while other decoding‑time fixes do not.

By Shamanthak Hegde, Xiangrui Liu, Maitreya Patel, Yezhou Yang
arXiv Machine Learning
Sep 25

Continuous Online Fault Detection for Mobile Robots via Adaptive Edge Models

The paper presents a Teacher-Student distillation framework for continuous online fault detection in mobile robots. An offline foundation model (TSPulse) generates pseudo‑labels from augmented time‑series data, while a lightweight MiniRocket Student, enhanced with a Recursive Least Squares estimator, performs real‑time inference with a 4.30 ms CPU latency. The Student adapts online to domain shifts, improving VUS‑PR scores from 0.26 to 0.75 and uses an uncertainty‑guided active learning strategy to request minimal operator interventions.

By Jordan Levy, Nicolas Verstaevel, Vincent Talon, Benoit Gaudou