Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 12

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

Vision2Code is a new benchmark for image-to-code generation that covers 15 domains and 539 test examples without relying on reference source code. It evaluates generated programs by executing them and comparing rendered outputs to the original images using domain‑specific rubrics, yielding scores that correlate better with human judgments than generic metrics. The study shows domain variability in model performance and demonstrates that the scoring method can improve fine‑tuning of models like Qwen3.5‑9B and Phi‑4‑MM via rejection sampling.

By Ajay Vikram Periasami, Junlin Wang, Bhuwan Dhingra
arXiv Machine Learning
Oct 12

MetaJoin: Adaptive Join Selections for Heterogeneous Tables

MetaJoin is a label‑free framework that adaptively selects join methods for heterogeneous tables by first choosing an initial specialist based on input evidence and then validating or repairing the result using specialist‑specific output evidence. It introduces MNN Join, a transformation specialist that finds string transformations by maximizing mutual‑nearest‑neighbor coverage and uses reciprocal validation. Evaluated on 670 datasets, MetaJoin achieves a macro row‑F1 of 0.817, surpassing a benchmark oracle and outperforming other label‑free policies by over 0.13, while MNN Join dominates all transformation baselines on 609 transformation‑heavy datasets.

By Ning Wang, Sainyam Galhotra
arXiv Machine Learning
Oct 12

Bayesian dynamic scheduling of multipurpose batch processes under incomplete look-ahead information

The paper introduces a Bayesian dynamic scheduling method for multipurpose batch processes that operates under incomplete look‑ahead information. By learning a Bayesian Network from the probability distribution of disturbances, the method updates posterior distributions online and guides rescheduling decisions. Compared with a periodic rescheduling strategy, the approach yields statistically better long‑term costs and reduced system nervousness on benchmark problems.

By Taicheng Zheng, Dan Li, Jie Li
arXiv Machine Learning
Oct 12

Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

The paper examines whether deep ensembles effectively capture uncertainty in message‑passing graph neural networks. Across eight diverse datasets, ensembles only marginally outperform single models, mainly by reducing optimization noise rather than improving uncertainty estimates. The authors identify an epistemic collapse—independently trained networks converge to similar predictions—due to the smoothing bias of message passing, and provide a theoretical bound linking this collapse to graph topology and Dirichlet energy.

By Pedro C. Vieira, Viacheslav Borovitskiy, Pedro Ribeiro
arXiv Computer Vision
Oct 12

SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

SheafPrune introduces a sheaf‑guided graph propagation method for pruning visual tokens in multimodal large language models. It constructs modality‑specific maps and a sparse text‑visual graph to transfer relevance, then propagates and degree‑corrects scores on a visual graph before selecting a hard top‑k subset. Evaluated across four backbones, ten benchmarks, and seven pruning ratios, SheafPrune consistently outperforms baselines, achieving up to a 5.15× prefill speedup and 84.0% FLOPs reduction on LLaVA‑NeXT‑8B while retaining 10% of tokens.

By Xiaoyang Ma, Chi Wang, Hong Sun
arXiv Machine Learning
Oct 12

RPP: A Certified Poisoned-Sample Detection Framework for Backdoor Attacks under Dataset Imbalance

The paper investigates how dataset imbalance increases vulnerability to backdoor attacks and degrades existing defenses. It introduces Randomized Probability Perturbation (RPP), a black‑box framework that detects poisoned samples using only model output probabilities, providing provable detectability guarantees and a bound on false positives. Experiments on five benchmarks and multiple attacks show RPP outperforms state‑of‑the‑art defenses, especially under class imbalance.

By Miao Lin, Feng Yu, Rui Ning, Lusi Li, Jiawei Chen, Qian Lou, Mengxin Zheng, Chunsheng Xin, Hongyi Wu
arXiv Computer Vision
Oct 12

Beyond the Snowflakes: Disentangling Weather Effects from Confounding Meta-Factors in LiDAR Object Detection

The paper investigates how snow affects LiDAR-based object detection, finding that detection accuracy actually improves with higher snow intensity. This counter‑intuitive result is explained by an increase in object point count due to snow accumulation, which outweighs the noise introduced by falling snow. The authors argue that broad weather severity bins can mislead robustness assessments unless atmospheric noise and surface accumulation effects are disentangled.

By Yiming Yang, Jeremy P. Bos
arXiv Machine Learning
Oct 12

Behind-the-Meter Photovoltaic Visibility from Net Smart Meters: Deployment Regimes and Operational Insights

The paper investigates how the operational value of deep‑learning PV disaggregation depends on deployment regimes—whether the model serves known customers, new customers, future periods, or new regions. Using Ausgrid smart‑meter data and cross‑dataset adaptation to Pecan Street, the authors benchmark several deep sequence architectures and find that deployment regime explains more error variation than model choice. Fine‑tuning on a small number of target‑region households can significantly reduce transfer error, and the benefits of long input windows vary by use case.

By Jack Zheng, Hao Wang
arXiv Machine Learning
Oct 12

A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges

The paper presents a comprehensive framework for comparing parameter‑efficient fine‑tuning (PEFT) methods by treating each as a smooth map from a small parameter space into weight space, with a pretrained checkpoint as the base point. It identifies three key invariants—image near the base point, fibre dynamics, and base change—that explain expressivity, training dynamics, and initialization behavior. The authors prove numerous results about common PEFT techniques (e.g., LoRA, Householder adaptation) and classify 382 methods from 138 papers using seven coordinate descriptors.

By Xiaoyu Li, Zhizhou Sha, Chiwun Yang, Dai Shi
arXiv Machine Learning
Oct 12

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.

By Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng
arXiv Machine Learning
Oct 12

Low-Rank Kolmogorov-Arnold Generator Flows for Interpretable Operator Learning

The paper introduces KAFlow, a low‑rank Kolmogorov‑Arnold flow that learns the generator of physical operators as a compact symbol composed of univariate Kolmogorov‑Arnold functions. By fixing the gauge of these functions, integrating dynamics with exact exponential integrators, and automatically distilling closed‑form expressions, KAFlow achieves markedly lower out‑of‑distribution loss and higher precision on synthetic benchmarks, long‑horizon quantum dynamics, and variable‑order fractional operators. The method recovers operator coefficients to within 10⁻⁴ and discovers complex dispersion relations that sparse regression cannot capture.

By Minjong Cheon
arXiv Machine Learning
Oct 12

Tensor-based Fusion of Multi-modal Data for Predicting Cancer Risk Levels

The paper introduces an unsupervised machine learning framework that fuses heterogeneous genomic and clinical data using tensor decomposition and clustering to predict cancer risk levels. By extracting latent, biologically meaningful features through tensor decomposition, the approach outperforms unimodal feature sets. Experiments on LinkedOmics data of 11,158 patients across 32 cancer types show that the tSVD+GMM configuration consistently achieves superior performance on clinical benchmarks such as tumor purity, pathological stage, and survival outcomes.

By Ali Anaissi, Yuanzhe Jia, Qishun Peng, Weidong Huang, Ali Braytee, Junaid Akram
arXiv Machine Learning
Oct 12

Trainable Quantum-Native Language Models: Dissipative Memory, Capacity Limits, and Hardware Validation

The paper introduces byte‑level autoregressive language models that use a small quantum register as hidden state, with token‑conditioned quantum channels and Born‑rule measurements providing next‑byte probabilities. Unitary dynamics alone fail to retain memory beyond one token, but adding learned dissipative gating improves test negative log‑likelihood and key‑value recall up to a limited capacity. Experiments on IBM Heron hardware show measurable quantum memory signals and autoregressive byte generation, though the models remain classically simulable and do not outperform classical n‑gram baselines.

By Deniz Yunus G\"o\u{g}\"u\c{s}
arXiv Machine Learning
Oct 12

When Solvers Get Moody: Quantifying and Predicting Performance Variability in Mixed Integer Programming

The paper introduces the Weighted Variability Score (WVS), a metric that aggregates dispersion in runtime, optimality gap, LP iterations, and branch‑and‑bound nodes across repeated runs of MIP solvers. Experiments on 100 randomized runs of MIPLIB 2017 instances show a right‑skewed distribution, with many instances stable but some exhibiting large variability not explained by basic properties. A tree‑based predictor using static and root‑level features achieves a mean absolute error of 0.058 and Spearman rank correlation of 0.65, with root degeneracy, matrix structure, coefficient ranges, and presolve reductions identified as key drivers of high WVS.

By Mourchid Adegbindin, Brenda Dietrich, Jamol Pender, Andrea Lodi
arXiv Machine Learning
Oct 12

Golden Anchor for Video Generation

The paper introduces Golden Anchor Learning (GAL), a framework that optimizes text-to-image (T2I) generators using feedback from downstream image‑to‑video (I2V) models to produce ‘golden anchors’ that improve video quality. Experiments show that GAL consistently raises VBench scores by 2.97%–8.46% and enhances motion quality across four I2V models, even when trained only on HunyuanVideo‑1.5 renders.

By Jinyuan Xu, Shitong Shao, Lichen Bai, Zikai Zhou, Guoxi Huang, Nantheera Anantrasirichai, Zeke Xie
arXiv Machine Learning
Oct 12

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Sonic-AMG is a novel framework that generates human motion in real time from spatial audio, the first to do so in a streaming manner. It encodes acoustic content and body-relative source geometry as separate tokens, then uses a causal transformer with a flow head to produce continuous motion latents that a causal decoder turns into motion segments. The system introduces rehearsal training and spatial feedback to mitigate training‑inference mismatches, achieving state‑of‑the‑art FID and R‑precision on the SAM benchmark while enabling coherent, real‑time reactions to sound events and source locations.

By Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura, Srinath Sridhar
arXiv Machine Learning
Oct 12

A Scalable Learned Iterative Solver for Nonlinear MPC via Temporal Decomposition

The paper presents a scalable learned iterative solver for nonlinear model predictive control (NMPC) that removes dependence on the prediction horizon by exploiting temporal decomposition in the network architecture. It introduces a closed‑loop resampling strategy to align training with deployment, achieving up to four‑fold speedups over IPOPT while maintaining comparable control performance and avoiding constraint violations. The method outperforms the previous monolithic LISCO approach on a CSTR cascade benchmark, scaling to larger problems without increasing network size.

By Lukas L\"uken, Sergio Lucia
arXiv Machine Learning
Oct 12

PPG-Bench: A Unified Benchmark for PPG Foundation Models

PPG-Bench is an open‑source benchmark that evaluates five publicly available photoplethysmography (PPG) foundation models across nine datasets and 26 tasks, covering cardiovascular state estimation, activity, sleep, screening, and identification. The benchmark tests models under four downstream regimes—linear probing, random forests, frozen MLP probes, and end‑to‑end finetuning—while also comparing them to three controls: a random/constant baseline, a deterministic pulse‑rate‑variability feature set, and a generic time‑series foundation model (MOMENT). Results show that over a third of task‑regime combinations yield no advantage for any PPG foundation model over the baseline, and in most cases no PPG‑specific model outperforms the simple controls, indicating that current PPG foundation models lack generalised, transferable representations.

By Arvasu Kulkarni, Mahir Jain, Aditya Ray Mishra, Sandeep Singh, Siddharth Panwar
arXiv Machine Learning
Oct 12

SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark

SeoulMMOD is a new large‑scale public benchmark that provides three years of hourly origin‑destination flow data for six urban travel modes across 25 districts and 426 sub‑districts in Seoul, along with travel‑time, travel‑distance, calendar, rainfall, and point‑of‑interest signals. The dataset enables city‑wide multimodal mobility analysis and supports the evaluation of 14 forecasting baselines under a common protocol. The benchmark reveals that models effective at the district level may not scale to sub‑district forecasting and that naive joint multi‑mode training does not always improve accuracy.

By Taeyoung Yu, Seonbin Jo, Jiwon Kim, Junyoung Byun
arXiv Machine Learning
Oct 12

CV-OPD: Contrastive Vision On-policy Distillation

The paper introduces Contrastive Vision On‑policy Distillation (CV‑OPD), a method that enhances on‑policy distillation for vision‑language models by adding a contrastive divergence loss and a discrepancy‑aware gating mechanism. CV‑OPD contrasts teacher distributions from positive and negative visual crops, forcing the student to rely on genuine visual evidence rather than shortcuts. Experiments on fine‑grained visual benchmarks show that CV‑OPD reduces shortcut learning and improves visual reasoning across existing VLMs.

By Haowei Wang, Hao Lin, Qi Zhu, Jiayi Ji, Deyi Ji, Liujuan Cao, Rongrong Ji