Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 17

REPAIR: Resolving Long-Tail Confusion in Scientific Retrievers via Fact-Verified Iterative Refinement

REPAIR is a self‑evolving data augmentation framework designed to improve scientific dense retrievers by addressing long‑tail concept gaps and fact sensitivity. It iteratively generates training data through diagnosis of long‑tail concepts, API‑guided evidence expansion, and hard negative mining, thereby grounding retrieval in factual reality. Experiments show that REPAIR outperforms 19 strong baselines across nine materials science and biomedical benchmarks.

By Yerim Oh, Gunhee Kim
arXiv AI
Sep 17

Principled Koopman Representations with Kalman Inference for Efficient Time-Series Prediction

The paper introduces K$^2$SVD, a method that learns the leading singular functions of the Koopman operator by optimizing a Hilbert-Schmidt objective, producing a low‑rank, interpretable Koopman representation with a compact latent space. In this space, temporal evolution is modeled with a linear Gaussian state‑space model and inference is performed via Kalman filtering to reduce noise accumulation in multi‑step predictions. Experiments demonstrate that K$^2$SVD outperforms state‑of‑the‑art methods on multiple datasets, achieving faster prediction speeds and lower computational cost.

By Ruiquan Li, Yuheng Bu
arXiv AI
Sep 17

RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos

RoboVAD is a large-scale benchmark for video anomaly detection in robotic arm manipulation, featuring cross-domain evaluation where both tasks and anomaly types are unseen during training. The dataset challenges existing VAD methods, with state-of-the-art approaches, including a new method tailored for robotic arms, still achieving less than 70% micro-averaged frame-level AUC in the hardest setting. The authors provide the dataset and code publicly for further research.

By Alexandru-Bogdan Dura, Sebastian Balmus, Radu Tudor Ionescu
arXiv AI
Sep 17

I code or AI code: A comparative evaluation of AI-rated scores in classroom observations

The study evaluated whether a large language model (GPT‑5) could score teacher‑child interactions in early childhood classrooms using the Classroom Assessment Scoring System (CLASS) framework, comparing its results to human raters. Across 87 video‑recorded observations from 38 classrooms in Hong Kong, AI scores converged most closely with human ratings in the Emotional Support domain, especially the Quality of Feedback dimension, while diverging more in procedural or context‑dependent areas such as Classroom Organization and Instructional Support. The findings suggest that transcript‑based AI scoring can serve as a preliminary screening tool to aid teacher reflection, but it is not yet reliable enough to replace trained observers for full CLASS evaluations.

By Y. Fong, J. Xiang, T. Y. D. Chan, K. Lee, E. Y. H. Lau
arXiv Machine Learning
Sep 17

Structural Inference under Hidden Agents

The paper introduces SIHA, a method for jointly recovering hidden-agent trajectories and their interaction structures in multi-agent dynamics. It tackles the circular dependency between trajectory reconstruction and structural inference by combining a structure‑agnostic initialization with iterative, structure‑guided refinement using Neural Relational Inference and multi‑strength structural attention. Experiments on benchmark dynamical systems and motion‑capture data show that SIHA improves both visible‑to‑visible structural inference and hidden‑state reconstruction, leading to better future predictions.

By Zhongben Gong, Xiaoqun Wu, Mingyang Zhou, Hui Huang
arXiv Machine Learning
Sep 17

WaveTLM: Reliable Time-Series Language Modeling through Task Compilation

WaveTLM introduces a unified compiler‑executor framework that transforms natural‑language requests into typed task states and reliable numerical or structured outputs for time‑series tasks. The authors present ExecTS‑QA, a benchmark covering forecasting, imputation, classification, anomaly detection, and waveform analysis, and show that a single WaveTLM checkpoint achieves 99.40% contract‑valid coverage versus 37.83% for a string‑first baseline while maintaining balanced predictive performance. Additional evaluations on SciTS, TSQA, IRTS‑ToolBench, and ARFBench demonstrate the model’s transferability.

By Jiahui Chen, Bingke Zhu, Hongyu Pan, Yingying Chen
arXiv Machine Learning
Sep 17

HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometries

HiLNO is a hierarchical latent neural operator that builds a fine‑to‑coarse‑to‑fine latent space and incorporates multi‑scale supervision and anisotropic Gaussian attention to preserve spatial information in PDE solutions with multiscale structures. The hierarchy reduces information loss during compression, while multi‑scale supervision aligns intermediate predictions with downsampled targets, and anisotropic attention facilitates feature transfer across scales. Experiments on standard PDE benchmarks and a large‑scale automotive aerodynamics task show that HiLNO achieves competitive accuracy while cutting parameter count by 84.4% and FLOPs by 69.2% compared with LinearNO, and it generalizes effectively to unseen spatial resolutions.

By Zhicheng Hu, Jiacheng Li, Min Yang
arXiv Machine Learning
Sep 17

Learning from Distributed Eyes: Leveraging Collaborative Perception for Automated Model Adaptation

The paper introduces LDE, a framework that uses collaborative perception (CP) to generate high‑quality pseudo‑labels for unsupervised model adaptation in autonomous driving. It tackles communication limits, field‑of‑view mismatches, and label unreliability through selective feature sharing, FoV filtering, and curriculum learning. Experiments on 3D object detection show LDE surpasses pre‑trained models and existing adaptation methods.

By Yanan Ma, Yihang Tao, Zhengru Fang, Zihan Fang, Yiqin Deng, Xianhao Chen, Yuguang Fang
arXiv Machine Learning
Sep 17

vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes

The paper introduces vidax, an open‑source JAX/Flax inference engine that enables video generative models to run on accelerator meshes such as Cloud TPU pods. Vidax includes a zero‑copy PyTorch‑to‑JAX weight translator and supports a wide range of spatiotemporal architectures—including Diffusion Transformers, Mixture‑of‑Transformers, 3D VAEs, and text encoders—without requiring PyTorch in the execution path. The framework unifies 1D tensor parallelism with DeepSpeed‑Ulysses sequence parallelism on a single JAX sharding mesh, incorporates TPU flash‑attention kernels, and implements per‑layer weight offloading to handle reference resolutions that exceed single‑device memory, with benchmarks on TPU v4‑8 hardware and documentation of real‑world numerical bugs.

By Congyue Deng
arXiv Machine Learning
Sep 17

Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

The paper investigates how to balance retaining past experience versus learning from new data when robot dynamics change. It introduces two metrics—change magnitude and age‑staleness AUC—to quantify when older transitions are helpful or harmful. Experiments on locomotion tasks and real‑world perturbations show that the optimal replay strategy depends on the size of the dynamics shift and the evolution of the system over time.

By Everest Yang, Skye Thompson, George D. Konidaris
arXiv Machine Learning
Sep 17

Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors

The paper introduces the Behavior-to-Value (B2V) task, which seeks to identify consumer values from e-commerce behavioral trajectories. It presents the E-commerce Consumption Value Taxonomy (ECVT) and the B2V-Bench dataset, derived from anonymized Taobao logs and covering 25 purchase behaviors with associated value orientations. A new model, B2V-Verifier, is proposed to improve value measurement accuracy, achieving a 34% boost in multi-label classification over strong LLM baselines.

By Peixuan Hou, Bin Chen, Li He, Jian Xu, Bo Zheng, Xiuli Ma, Guojie Song
arXiv Machine Learning
Sep 17

Reinforcement Learning for Real-Time Vision-Language-Action Policies

The paper presents Real‑Time EXPO‑FT, a reinforcement learning framework that fine‑tunes large Vision‑Language‑Action models for real‑time robotic control. It separates slow, expressive action generation from fast, reactive edits, allowing a lightweight policy to adjust actions based on the latest observation. Experiments on the Kinetix benchmark and four dynamic real‑world tasks show that Real‑Time EXPO‑FT achieves superior performance, improving policy success rates from 42% to 97% with only ten minutes of online data and no human intervention.

By Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn
arXiv Machine Learning
Sep 17

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

TTM-Bench is a framework designed to benchmark text-to-music systems by establishing a common protocol for reproducible evaluation. It measures performance along two axes: musical-content alignment—assessed through semantic, genre, and musical-descriptor agreement scores against a shared musical specification—and computational efficiency, which includes generation latency, real-time factor, resource usage for local models, and cost for hosted services. A preliminary case study using TTM-Bench shows that higher alignment does not necessarily mean lower computational demands, underscoring the need for distinct, interpretable metrics.

By Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi
arXiv Machine Learning
Sep 17

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

The paper argues that current fallacy-detection benchmarks are flawed because they pair fallacy classes with a broad "valid" or "none" class that includes many correct arguments, leading to misleadingly low false‑positive rates. By constructing scheme‑matched negatives—correct arguments that use the same argumentation scheme as the fallacy—the authors show that false‑positive rates rise dramatically, indicating that classifiers are learning to recognize schemes rather than detecting fallacies. The study releases these scheme‑matched examples as Scheme Foils and cautions that reported false‑positive rates should not be trusted until the valid class is audited for scheme coverage.

By Navyansh Singh, Animesh Pathak, Aarav Singh
arXiv Machine Learning
Sep 17

Topology-enhanced machine learning for speech signal processing

The paper introduces TopCap, a topology‑enhanced method for extracting features from speech time series. TopCap captures fine structural details, such as vibrations, that traditional spectral analysis may miss. When applied to classifying voiced versus voiceless consonants, TopCap matches neural network accuracy, and when combined with neural networks it improves robustness to noise, accuracy, stability, convergence, and interpretability.

By Pingyao Feng, Qingrui Qu, Haiyu Zhang, Siheng Yi, Zhiwang Yu, Zeyang Ding, Yifei Zhu
arXiv Computation and Language
Sep 17

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

The paper introduces a benchmark for evaluating open‑source instruction‑tuned large language models (LLMs) on key‑value pair extraction from documents under both clean‑text and noisy OCR conditions. It tests decoder‑only models such as Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek on FUNSD, CORD, and SROIE datasets, using OCR outputs from PaddleOCR, EasyOCR, and Tesseract. Results show that while modern LLMs perform well on high‑quality text, their performance drops sharply with OCR noise, and the main determinants of success are semantic reasoning and textual fidelity, with larger models offering diminishing returns under noisy inputs.

By Zahra Anvari, Vassilis Athitsos
arXiv Machine Learning
Sep 17

Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representations

The paper introduces a biology-informed heterogeneous graph representation that models retinal vessel segments, intercapillary areas, and the foveal avascular zone to predict diabetic retinopathy stages from OCTA images. This graph-based approach reframes staging as a graph-level classification task solved with a graph neural network, achieving AUC-ROC values up to 84% and outperforming biomarker-based classifiers, CNNs, and vision transformers. The method also provides detailed, interpretable explanations by precisely localizing abnormal vessels and non-perfusion areas.

By Laurin Lux, Alexander H. Berger, Maria Romeo Tricas, Richard Rosen, Alaa E. Fayed, Sobha Sivaprasada, Linus Kreitner, Jonas Weidner, Martin J. Menten, Daniel Rueckert, Johannes C. Paetzold