Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
Oct 28

How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop

The paper introduces a denoised counterfactual protocol to accurately measure the selectable headroom of world action models (WAMs) and uses these measurements to build a fail‑gated selector. Across four WAMs and two benchmarks, the study shows that the standard estimate overstates selectable headroom, that structured action perturbations can add up to 16 points of headroom, and that imagined futures retain candidate‑outcome information without loss. The selector improves closed‑loop task success from 76.9 % to 81.6 % on held‑out training episodes and from 62.6 % to 69.8 % on unseen tasks.

By Haojia Gao, Yue Cao, Yumin Liu, Haohua Que, Mingkai Liu, Guanqiu Guo, Zhuo Zou, Bokui Chen, Xueqian Wang
arXiv Computer Vision
Oct 28

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

The paper investigates how multimodal large language models decide to refuse answering visually unanswerable questions. It examines three control mechanisms—instruction phrasing and placement, post‑training methods, and internal activation interventions—across three models on the MoHoBench benchmark. The study finds that refusal behavior is highly sensitive to instruction details, can be amplified by post‑training (sometimes leading to mixed refusal and answer attempts), and can be modestly altered by activation interventions without causing indiscriminate refusals.

By Ananth K Suresh, Aveen Dayal, Gouthaman KV, Aditya Pola, Vineeth N Balasubramanian
arXiv Computer Vision
Oct 28

Single-Step Bidirectional Unpaired Image Translation Using Implicit Bridge Consistency Distillation

The paper introduces Implicit Bridge Consistency Distillation (IBCD), a method that enables single-step, bidirectional unpaired image translation without adversarial loss. IBCD uses a diffusion implicit bridge model that connects PF-ODE trajectories between distributions, parametrized on a signed time axis to select the target domain, allowing one network to translate in both directions in a single step. The authors also propose distribution matching for consistency distillation and an adaptive weighting scheme based on distillation difficulty, achieving state‑of‑the‑art performance on benchmark datasets and extending to brain MRI contrast translation.

By Suhyeon Lee, Kwanyoung Kim, Jong Chul Ye
arXiv Computer Vision
Oct 28

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

LESSViT introduces a sensor‑flexible Vision Transformer for hyperspectral imagery that uses a low‑rank structured attention mechanism (LESS Attention) to efficiently model joint spatial‑spectral interactions. By reducing the complexity from ≠O(N^2C^2) to ≠O(rNC) and adding channel‑agnostic patch embedding with wavelength‑aware positional encoding, it supports flexible spectral inputs. A hyperspectral masked autoencoder (HyperMAE) with decoupled spatial‑spectral masking further enables robust pretraining, and experiments on the SpectralEarth benchmark show improved robustness to spectral shifts while maintaining competitive in‑distribution performance.

By Haozhe Si, Yuxuan Wan, Yuqing Wang, Minh N. Do, Han Zhao
arXiv Machine Learning
Oct 28

Differentiable FEM versus PINN for pavement layer inverse analysis: enforcing physics as hard or soft constraints

The paper compares physics‑informed neural networks (PINNs) and differentiable finite element methods (DiffFEM) for inverse analysis of multilayer pavement systems using falling weight deflectometer data. PINNs struggle to recover layer moduli due to sharp domain discontinuities, and their performance is highly sensitive to loss weighting, network architecture, and measurement noise. DiffFEM consistently delivers more accurate, stable, and computationally efficient inversion results by enforcing physics as a hard constraint.

By Yongjin Choi, Hyeonbin Moon, Seunghwa Ryu
arXiv Computer Vision
Oct 28

VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation

VibeVideoBench is a benchmark that evaluates multimodal agents capable of generating videos by writing code to render each frame and sound. The benchmark contains 239 self‑contained tasks in English and Chinese across 12 categories, requiring agents to produce a finished MP4 with sound within one hour without using any pre‑trained image or video generation model. Evaluation is hierarchical, using program checks, 13k task‑specific rubric checks, and five quality dimensions scored by three VLM judge panels, and it shows that GPT‑6 Astra and Claude Opus 5.5 perform best among nine frontier agents.

By Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Han Li, Wentao Zhang, Weiran Xu, Kongming Liang, Zhanyu Ma
arXiv Computer Vision
Oct 28

Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

DreamTacVLA is a Vision‑Language‑Action framework that incorporates high‑resolution tactile images alongside wrist‑camera and third‑person vision to enable robots to anticipate and react to contact dynamics. It uses a Hierarchical Spatial Alignment loss to align tactile tokens with visual counterparts and a tactile world model to predict future tactile signals, trained on a hybrid dataset from digital twins and real experiments. The approach achieves up to 95% success on contact‑rich manipulation tasks, outperforming existing VLA baselines.

By Guo Ye, Zexi Zhang, Xu Zhao, Shang Wu, Haoran Lu, Shihan Lu, Han Liu
arXiv AI
Oct 28

UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

UnityBench-Mini3D is a benchmark comprising 77 Unity project-level tasks organized into seven expert-defined skill categories and three difficulty tiers. Each task pairs a complete Unity project with natural-language requirements and is evaluated using 1,178 held-out, human-authored milestone tests that assess scene structure, component configuration, and runtime behavior. The benchmark provides deterministic partial credit, localizes failures, and distinguishes partial implementations from complete solutions without relying on subjective judges, achieving an overall pass rate of 57.7% across 21 models and 9 coding harnesses.

By Yijun Zhou, Minghao Shao, Ke Zhang, Renkai Yuan, Hongbo Yuan, Bin Hu, Bozhao Qian, Hongxia Shen, Yanhong Liu, Yuting Chen, Yuntao Shi, Zhongxuan Xu, Shuo Mao, Haijia Zhou, Zhendong Li, Tao Zhang, Chenggang Li, Yuxuan Wang, Zinan Zheng, Calvin Zhang, Jia Li, Heng Chang, Jie Feng
arXiv Machine Learning
Oct 28

Performance of Neural Operator Baselines on 'The Well' Dataset

The Well is a 15‑terabyte benchmark covering 16 physical simulation domains, aimed at evaluating machine learning surrogate models for complex systems. Baseline neural operator results were presented as illustrative, with minimal hyperparameter tuning and a strict 12‑hour per‑run limit. A brief hyperparameter search on the Tucker‑factorized Fourier Neural Operator, using only a single input timestep, lowered VRMSE below most published baselines on 15 of 17 tasks, showing that modest tuning can dramatically improve performance.

By Yash Pincha, Andy Dimnaku, Andrew Chiang, Adarsh Ganeshram, Valentin Duruisseaux, Anima Anandkumar
arXiv Machine Learning
Oct 28

INTRA: Interleaved Non-contiguous Token spaRse Attention

The paper introduces INTRA, a hardware‑aware static sparse attention mechanism that interleaves non‑contiguous token groups to enable efficient long‑range communication while reducing quadratic cost. INTRA uses Scatter and Gather layers to create short mediated paths across layers, and a FlashAttention‑style kernel computes structured indices at runtime for dense packing of selected tokens. Experiments on an A100 GPU show significant speedups—up to 8.4× over MoBA and 10.4× over density‑constrained NSA—while maintaining or improving image quality and long‑context performance on models like FLUX.1‑dev and LLaMA‑3.1‑8B.

By Yuxuan Xia, Wenbo Lu, Shaoyi Zheng, Shengjie Wang
arXiv Machine Learning
Oct 28

OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding

OneEEG is a unified signal‑visual multimodal language model designed for multi‑task EEG understanding. It jointly learns from numerical EEG data and visual EEG renderings, employing a task‑aware Mixture of Projectors to align diverse EEG features with a large language model’s embedding space. The authors introduce the OneEEG‑Instruct dataset for instruction tuning and the OneEEG‑Bench benchmark to evaluate performance, demonstrating that existing multimodal LLMs struggle with EEG inputs while OneEEG outperforms them across various tasks.

By Jiacheng Zhang, Xueyi Zhang, Xin Lu, Siqi Cai, Haizhou Li
arXiv AI
Oct 28

Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception

The paper introduces EvidenceConnect, a data‑synthesis framework that generates training examples by zooming into related image regions to create questions requiring comparison, aggregation, or chaining of visual evidence. By retaining only reference answers agreed upon by independent vision‑language models and discarding questions that survive region ablation, the authors train base VLMs with RLVR to produce Zooming without Zooming v2 (ZwZ2). Although the synthesized data are not benchmark‑specific, ZwZ2 shows improvements across fine‑grained and spatial perception, embodied spatial reasoning, multimodal long‑context understanding, multimodal agent tasks, and general visual reasoning, demonstrating that strengthening the shared capability of connecting evidence yields more generalizable multimodal perception.

By Lai Wei, Yatai Li, Liangbo He, Jun Lan, Renjie Gu, Linhan Cao, Junkai Lin, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, Zhuosheng Zhang, Weiran Huang
arXiv AI
Oct 28

Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

The paper introduces Fisher-Shaped EWC (FS‑EWC), a replay‑free continual learning method that extends Elastic Weight Consolidation by controlling Fisher sensitivity over a neighborhood of future updates. FS‑EWC minimizes the worst‑case empirical Fisher information trace during task learning, favoring solutions robust to bounded parameter displacement, and uses trajectory‑based model merging to construct the retained solution from multiple optimization states. Experiments on several benchmarks demonstrate that FS‑EWC outperforms standard EWC, achieving a better stability–plasticity balance and higher tolerance to parameter drift.

By Liying Wang, Simone Magistri, Joost van de Weijer
arXiv Machine Learning
Oct 28

CLARiTy: A Vision Transformer for Multi-Label Classification and Weakly-Supervised Localization of Chest X-ray Pathologies

CLARiTy is a vision transformer model that jointly performs multi‑label classification and weakly‑supervised localization of chest X‑ray pathologies. It uses class‑specific tokens to produce attention maps and a SegmentCAM module to refine foreground segmentation with anatomical priors. Trained on NIH ChestXray14 image‑level labels and distilled from a ConvNeXtV2 teacher, CLARiTy achieves competitive classification and strong localization, especially for small pathologies, outperforming prior methods by up to 50.7% relative Macro IoU accuracy.

By John M. Statheros, Hairong Wang, Richard Klein
arXiv Computation and Language
Oct 28

The State and Fate of Multilingual, Contextual Evaluation in the NLP World

The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.

By Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram
arXiv Computation and Language
Oct 28

Hierarchical Low-Rank Reparameterization for Parameter-Efficient

The paper introduces a hierarchical low‑rank reparameterization technique that reorganizes the trainable parameters of parameter‑efficient fine‑tuning (PEFT) methods into an approximately square matrix and applies recursive low‑rank decomposition. This multi‑level compression drastically reduces the number of trainable parameters while preserving strong task performance. Experiments on LLaMA‑3.1‑8B and Qwen2.5‑7B show that, when combined with LoRA or DoRA, the method outperforms baseline PEFT approaches and remains competitive even after compressing the original parameter count by an order of magnitude.

By Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu
arXiv AI
Oct 28

MedConstBench: Benchmarking How Constitutions Steer LLMs in Medical Ethics

MedConstBench is a new medical‑ethics benchmark that treats a model’s deployment constitution as a controllable variable. It is organized around Beauchamp’s four principles and uses human‑calibrated peer review to score responses. The benchmark includes the MedDilemma dataset of clinical dilemmas and demonstrates that different constitutions influence LLM behavior in a dimension‑selective, model‑dependent way, leading to a proposed adaptive selector that chooses the best constitution per model.

By Yibo Yang, Zerui Huang, Haomiao Yang, Zixia Wang
arXiv Computer Vision
Oct 28

MVRCap: Towards Comprehensive Video Captioning via Hierarchical Multi-Dimensional Verifiable Rewards

MVRCap is a new video captioning framework that treats caption generation as a reinforcement learning problem guided by structured verification. It introduces a Fine‑Grained Verifiable Reward system built on three pillars—Hierarchical Concept Coverage, Factuality Verification, and Conciseness Constraint—supported by the newly created MVRCap‑QA and MVRCap‑SFT datasets. Experiments show that MVRCap outperforms existing methods on benchmarks such as VDC, CAPability, and VC4VG.

By Weihuang Lin, Xianing Chen, Zhibin Wang, Congzhi Zhang, Yiwei Ma, Jiayi Ji, Miao Chen, Jun Song, YuCheng, Bo Zheng, Xiaoshuai Sun, Rongrong Ji
arXiv Computer Vision
Oct 28

BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks

BiDex-100 is a simulation benchmark for bimanual dexterous manipulation, featuring 106 tasks across five capability axes—generalization, instruction following, limited demonstrations, long horizon, and memory. Each task includes an out‑of‑domain variant to test robustness, and the benchmark supports two hand embodiments with 8,080 teleoperated demonstrations totaling 6.25 million frames. The authors evaluate seven recent Vision‑Language‑Action models under a unified protocol, providing baseline performance for capability, robustness, and data efficiency.

By Yong Wang, Kecheng Zheng, Xing Zhu, Chao Xu, Yujun Shen, Shengze Cai