Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computer Vision
Oct 8

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

arXiv:2604.01569v2 Announce Type: replace Abstract: Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can...

By Jiahao Meng, Yue Tan, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Renrui Zhang, Xiangtai Li, Haodong Duan, Yunhai Tong, Ming-Hsuan Yang
arXiv Computer Vision
Oct 8

SpanVLA: Learning from Negative-Recovery Samples with Fast Action Bridging for Vision-Language-Action Model

arXiv:2604.19710v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models bring rich world knowledge and reasoning capabilities to autonomous driving. However, most driving VLAs are tra...

By Zewei Zhou, Ruining Yang, Xuewei Qi, Yiluan Guo, Sherry X. Chen, Tao Feng, Kateryna Pistunova, Yishan Shen, Lili Su, Jiaqi Ma
arXiv Computer Vision
Oct 8

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

arXiv:2610.04432v2 Announce Type: replace Abstract: Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on...

By Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang
arXiv Machine Learning
Oct 8

Cross-Agent Learning Signals Enable Coordinated Role-Decomposed LLM Training

The paper introduces DAC (Divide and Cooperate), a role‑decomposed training framework that separates evidence acquisition (searcher) and response generation (generator) in agentic search systems. DAC uses task‑specific external verification signals to provide cross‑verification rewards, allowing the generator to abstain when evidence is insufficient and to assign credit appropriately to each role. Experiments on seven QA benchmarks and two model backbones show that DAC consistently outperforms single‑agent and multi‑agent baselines, with gains attributed to coordinated improvements in both search and generation.

By Jaewan Park, Solbee Cho, Jay-Yoon Lee
arXiv Machine Learning
Oct 8

Walk fast but be careful: Understanding Parallel Sampling in Masked Diffusion

The paper introduces random walks on graphs as a controllable sandbox for studying parallel sampling strategies in masked diffusion models (MDMs). It shows that common parallel unmasking methods, such as lowest‑entropy scoring, are not universally superior to random sampling and that performance depends on the graph’s conditional dependence structure. The authors also propose training‑free bisection samplers that are provably exact for random walks and demonstrate their effectiveness on both graph‑walk tasks and pretrained MDMs for language generation and reasoning benchmarks.

By Vansh Bansal, Cholyeon Cho, Syamantak Kumar, Sujay Sanghavi, Purnamrita Sarkar
arXiv Machine Learning
Oct 8

Fluids You Can Trust: Property-Preserving Operator Learning for Incompressible Flows

The paper introduces a kernel‑based operator learning method that preserves key physical properties—such as incompressibility, periodicity, and turbulence—of the incompressible Navier–Stokes equations. By mapping input functions to expansion coefficients in a property‑preserving kernel basis, the method guarantees that predicted velocity fields analytically maintain these properties. The authors provide theoretical convergence guarantees, develop efficient computational techniques for large‑scale training, and demonstrate significant accuracy and speed improvements over neural operators on 2D and 3D flow benchmarks.

By Ramansh Sharma, Matthew Lowery, Houman Owhadi, Varun Shankar
arXiv Computer Vision
Oct 8

CADReasoner: Iterative Program Editing for CAD Reverse Engineering

CADReasoner is a model that iteratively refines its CAD reconstruction by comparing the input shape with the predicted mesh, outputting a runnable CadQuery Python program. It fuses multi‑view renders and point clouds and uses a scan‑simulation protocol to bridge realism gaps during training and evaluation. The approach achieves state‑of‑the‑art performance on DeepCAD, Fusion 360, and MCB benchmarks for both clean and scan‑sim tracks.

By Soslan Kabisov, Vsevolod Kirichuk, Andrey Volkov, Gennadii Savrasov, Marina Barannikov, Anton Konushin, Andrey Kuznetsov, Dmitrii Zhemchuzhnikov
arXiv Machine Learning
Oct 8

Child ASR Adaptation with Adult Retention: An Empirical Study

The study investigates how to adapt automatic speech recognition (ASR) models to child speech while preserving adult performance, focusing on Arabic and English data. It compares full fine‑tuning, LoRA, and post‑hoc weight‑space merging across encoder–decoder, encoder–CTC, and AudioLLM‑based ASR systems, using child and adult benchmarks. Results indicate that child adaptation is essential, especially for non‑native speakers, but direct adaptation can degrade adult accuracy; bilingual adaptation and weight‑space merging techniques help balance the trade‑off, with encoder–decoder fine‑tuning still yielding the best raw WER for bilingual models.

By Houssam Eddine-Othman Lachemat, Shammur Absar Chowdhury
arXiv Machine Learning
Oct 8

GeoPrior-Mamba: Structured Process Priors with Mamba for Fine-Resolution XCO2 Reconstruction

GeoPrior-Mamba is a new multi-directional Mamba framework that reconstructs fine‑resolution XCO2 fields from sparse satellite data by incorporating structured process priors derived from a language model. These priors encode biospheric uptake, ecosystem respiration, and anthropogenic emissions into deterministic tables, which are spatially instantiated with geographic, ecological, emission, and seasonal information and injected into the model via a lightweight knowledge adapter. On OCO‑2 data from 2018‑2020, the method achieves an RMSE of 0.81 ppm and an R² of 0.93, outperforming CAMS background interpolation and Trans‑XCO2, and shows faster convergence and consistency with ground‑based TCCON measurements.

By Zhao Meng, Yinan Cai, Siru Zhong, Juepeng Zheng, Haohuan Fu
arXiv Machine Learning
Oct 8

Towards AI-Generated Music Plagiarism Detection as a Version Identification Problem

The paper addresses the challenge of detecting plagiarism in music generated by AI, noting that traditional binary notions of plagiarism are insufficient due to nuanced musical elements. It evaluates whether existing music version identification models, originally designed for human covers, can transfer to human-to-AI plagiarism detection. The authors introduce COPYCAT, a large benchmark of 350,654 pairs derived from real plagiarism cases and augmented with generative re‑synthesis and DSP obfuscations, and demonstrate that a supervised embedding‑shift approach significantly improves detection performance over simple distance thresholding.

By Fotis Koutsikos, Ioannis Prokopiou, Spyridon Kantarelis, Vassilis Lyberatos, Pantelis Vikatos, Athanasios Aidinis, Themos Stafylakis, Athanasios Voulodimos, Giorgos Stamou
arXiv Machine Learning
Oct 8

Few Bits, One Law: Toward W2A4KV2

The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B. "whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."

By Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh, Steven Li
arXiv Machine Learning
Oct 8

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

arXiv:2610.10409v1 Announce Type: cross Abstract: General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities car...

By Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhuo Chen, Xierui Song, Juntao Dai, Boyuan Chen, Jiaming Ji, Fangneng Zhan, Mengkang Hu, Wei Xue, Yonggang Zhang, Han Hu, Tsung-Yi Ho, Yike Guo