Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 16

SPARCL: Spatially Adaptive Regional Coverage for Active Learning in Chest X-ray Analysis

SPARCL (Spatially Adaptive Regional Coverage for Active Learning) is a new active learning strategy for chest X‑ray lesion segmentation that balances predictive uncertainty with spatial coverage using a concave coverage function. It directs annotation effort toward under‑labeled regions, reverts to standard uncertainty sampling when spatial variation is absent, and guarantees near‑optimal greedy batch selection through a monotone submodular objective. The authors also propose Spatial Inconsistency and Performance Coverage metrics to reveal regional fairness, and demonstrate that SPARCL achieves 95% of fully supervised performance using only 24% of the data versus 29% for the best baseline.

By Juho Jung, Seongjae Kang, Mijoo Kim, Sungduk Cho, Yunkyoung Jun, Sunghoon Joo, Yoona Hwang
arXiv Computation and Language
Oct 16

PRISM: PRIvileged Sibling-guided Multi-Agent Topology Design

PRISM is a method for designing collaboration topologies in multi‑agent systems that use large language models. It leverages privileged information by having a teacher model read a sibling graph that also solved the same query, guiding a student model to match per‑decision targets during its own trajectory sampling. Experiments on six diverse benchmarks show that PRISM outperforms all evaluated topology‑design baselines, and matched‑teacher controls confirm the benefit of the privileged query information.

By Phuong Tuan Dat, Qi Li, Xinchao Wang
arXiv Computation and Language
Oct 16

APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

APRiL is an open‑ended benchmark designed to evaluate Chinese large language models’ social awareness. It comprises 2,196 evaluation instances across six categories, split into two tasks: Social Scenario Analysis, which tests understanding of social cues, mental states, and conflict structures, and Socially Aware Response, which checks whether models can generate appropriate, natural replies. Experiments on twelve models from six families show a consistent gap between understanding and response quality, indicating that stronger scenario analyses tend to improve response performance.

By Tian Lan, Han Zhang, Shanshan Wang, Yahan Yu, Jiang Li, Yugo Murawaki
arXiv Computation and Language
Oct 16

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

The paper introduces KBS, a Korean Benchmark Suite that includes six datasets—Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval—covering nine capabilities across 9,396 items. It emphasizes a non‑translation approach by reviewing, localizing, and redesigning tasks for Korean, and evaluates Korean‑developed, multilingual open, and closed API models under protocols that consider model access, prompt format, and scoring differences. The results highlight that Korean benchmark scores are influenced by task construction, scoring rules, prompting, and post‑training, suggesting that evaluations should report both final scores and the settings used.

By Yeonkyoung So, Jongmin Kim, Sungmok Jung, Gyuseong Lee, Sangho Kim, Jongyeon Park, Joonhak Lee, Seho Pyo, Gyeongje Cho, Seorin Kim, Jisoo Kim, Suyoung Park, Hyunji M. Park, Yelim Ahn, Yeongho Seo, Jaejin Lee
arXiv Computation and Language
Oct 16

MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

MindEval2 is a new evaluation framework for AI chatbots in mental health support, comprising Mental Health Quality Rubrics (MHQR) and the MindSim patient simulation harness. MHQR, developed with PhD-level psychologists, provides explicit, actionable criteria for assessing response quality, while MindSim offers realistic, longitudinal patient interactions with a memory component. The study evaluated 18 models, finding that although surface-level quality has improved, guidance calibration remains the main failure across all models.

By John Mendon\c{c}a, Daniel Tschernutter, Jos\'e Pombal, Areti Vassilopoulos, Catarina Botelho, Ignacio Sarasua, Maya D'Eon, Nuno Guerreiro, Ricardo Rei
arXiv Computation and Language
Oct 16

Explanation Generation for Contradiction Reconciliation with LLMs

The paper introduces a new task called reconciliatory explanation generation, where large language models (LLMs) must produce explanations that make contradictory statements compatible. It proposes reusing natural language inference datasets and new quality metrics for scalable evaluation. Experiments with 13 LLMs show limited success, especially when longer reasoning chains are involved, but reinforcement‑learning training improves performance on two narrative reasoning benchmarks.

By Jason Chan, Zhixue Zhao, Robert Gaizauskas
arXiv Computation and Language
Oct 16

What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

The paper introduces a weighted in-context influence (wICI) framework for selecting instruction‑tuning data, aiming to reduce redundancy and improve sample quality. It investigates what makes data effective from an in‑context perspective, explores the relationship between sample difficulty and influence, and evaluates how influence predicts tuning performance. Experiments on various models and benchmarks show that wICI consistently outperforms existing baselines under limited data budgets and reveals a negative correlation between sample difficulty and in‑context influence.

By Guangzeng Han, Xiaolei Huang
arXiv Computer Vision
Oct 16

DLOSAC: Joint Spatial-Geometric Image Estimation via Delaunay-Guided Local Optimization RANSAC

DLOSAC introduces a local optimization framework that jointly considers geometric consistency and spatial distribution for image geometric estimation. It constructs a Delaunay triangulation over candidate inliers, uses an edge-based consistency metric for geometry, and derives distribution descriptors from triangle areas and edges to assess inlier clustering. By combining these geometric and spatial regularizations with traditional point-level constraints, DLOSAC achieves state‑of‑the‑art performance across six real‑world datasets and three estimation tasks.

By Yu Liu, Meiqi Zhou, Zipei Fan, Guanyuan Feng, Xuan Song
arXiv Computer Vision
Oct 16

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.

By Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
arXiv Computer Vision
Oct 16

FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

FusionAudit introduces PaRMA, a pathway-conditioned robustness auditing framework for native multimodal vision‑language models. It evaluates two image‑level interventions—pixel perturbation and rendered‑text insertion—to expose architecture‑specific blind spots. The study applies diagnostic‑motivated attacks (MANTIS and Nat‑Evid) to 18 open‑weight VLMs, revealing that legacy single‑attack rankings are unstable and that PaRMA uncovers higher residual risk than standard benchmarks.

By Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan Lim
arXiv Computer Vision
Oct 16

Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration

The paper presents a clinically viable pipeline that uses routine intraoperative C‑arm fluoroscopy to derive per‑vertebra 6‑DoF poses, which are co‑registered to an RGB‑D camera via a multimodal fiducial array. This approach provides independent subsurface ground truth for validating markerless spinal tracking methods without adding radiation or altering standard surgical workflows. Ex vivo tests show an end‑to‑end accuracy of about 2 mm and 3.3°, and the framework is applied to evaluate two markerless registration strategies on real surgical data.

By Connor Daly, Salvatore Russo, Jinendra Ekanayake, Daniel S. Elson, Ferdinando Rodriguez y Baena
arXiv Computer Vision
Oct 16

Beyond Source Prompts: Towards Visually Grounded AI-Generated Video Quality Assessment

The paper introduces VGCap, a framework for assessing the quality of AI-generated videos that uses visually grounded captions instead of relying solely on source prompts. VGCap generates a global caption by minimally revising the source prompt with multi‑frame visual evidence and also produces prompt‑independent segment captions to describe localized temporal content. These captions jointly support evaluation of spatial quality, temporal quality, and text‑video alignment, and experiments on LGVQ and FETV datasets show VGCap outperforms existing baselines across all three dimensions.

By Kehao Zhou, Yixuan Li, Yuxuan Jiang, Fan Zhang
arXiv Computer Vision
Oct 16

Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach

The paper introduces a self‑supervised, geometry‑aware method that refines DINOv3 features with LoRA to better handle partial 3D shapes. By training on partial observations, the resulting descriptors outperform traditional and other vision‑foundation features in partial shape matching tasks. When incorporated into existing pipelines, the approach achieves state‑of‑the‑art results on both partial shape matching and left‑right prediction benchmarks.

By Viktoria Ehm, Dongliang Cao, Riccardo Marin, Daniel Scholz, Weikang Wang, Florian Bernard, Daniel Cremers
arXiv Computer Vision
Oct 16

VesperMUSE: Multi-Modal Fusion Vision Transformer for Image Semantic Segmentation Under Diverse Light Conditions

VesperMUSE is a multi‑modal vision transformer that enhances image semantic segmentation under varied lighting by combining physics‑informed low‑light adaptation, thermal‑guided low‑light image enhancement, and surface normal maps. The framework fuses RGB, thermal, and normal features through a modality‑agnostic self‑attention mechanism, enabling robust segmentation without architectural changes for different sensor setups. Extensive tests show that VesperMUSE recovers missing visual cues and achieves state‑of‑the‑art performance in extreme, unstructured environments.

By Raul Balmez
arXiv Computer Vision
Oct 16

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

The paper introduces a method to retrofit depth recurrence into pretrained vision‑language models, reducing the language backbone’s parameter count by pruning layers and looping over the remaining ones with linear adapters. A three‑stage recovery training—language‑only pretraining, vision‑language projector alignment, and full fine‑tuning—helps recover performance, with distillation from the non‑recurrent model further improving results. Experiments on TinyLLaVA‑1.4B show the retrofitted recurrent model outperforms the unpruned baseline while saving 23.3% of language parameters and performs better on reasoning‑oriented benchmarks.

By Madhav Iyengar, Andreas Geiger, Hilde K\"uhne
arXiv Computer Vision
Oct 16

MJev-Spatial: System One Spatial Intelligence via Native Attention Matching

MJev-Spatial is a multimodal decision model that unifies spatial intelligence tasks—choice, numerical, and point prediction—into a single classification framework. It treats each candidate as a class, scoring them with pretrained query and key projections without adding parameters, and achieves predictions in a single forward pass with no answer tokens. Trained on 12.3 M question‑answer pairs, it outperforms eight baseline models, ranks first on seven of twelve benchmarks, and runs 2.5× faster than the fastest autoregressive baseline.

By Luozheng Qin, Yuhan Zhu, Changhai Zhou, Qian Qiao, Jun Gao, Yuhua Zhou, Harry Zhang, Xiangyu Zeng, Yuxuan Liu, Kaijie Chen, Andrew Chen, Pony Ma
arXiv Computer Vision
Oct 16

Grounding Generative Verification in 3D Content

The paper introduces 3DVerifierBench, a benchmark covering ten tasks that evaluates 3D content generation for correctness, rejection explanations, and responsiveness to specification changes. It highlights a significant performance gap between humans and current models, and presents a scalable pipeline that generates over 14,000 training examples with contradiction explanations. The study also demonstrates how verification can improve 3D generation via candidate selection and recursive refinement, raising factual success rates dramatically.

By Shuanglin Li, Jiaying Lin, Shuquan Ye
arXiv Computer Vision
Oct 16

Gen4P: Video Generators are 4D Perceivers

Gen4P is a unified framework that transforms a pretrained video generator into a feed‑forward 4D perceiver, enabling recovery of 4D geometry and motion from monocular video without iterative denoising. It processes video latents in a single encoder pass and uses a lightweight cross‑attention decoder to predict requested 4D attributes, supporting flexible spatial sampling and continuous‑time queries. By integrating a frozen video‑depth LoRA as a geometry prior, Gen4P improves performance on 3D point tracking, video depth, and camera pose estimation, achieving state‑of‑the‑art or competitive results on benchmark tasks and demonstrating robust tracking of complex motion in in‑the‑wild videos.

By Yitong Xia, Zeren Jiang, Chuanxia Zheng
arXiv Computer Vision
Oct 16

Complementary Future Token Pruning for World Action Models

The paper introduces CF‑Prune, a training‑free method that prunes future‑frame tokens in pretrained World Action Models (WAMs) to reduce inference latency. By using instruction and action attention during the initial denoising pass, the framework identifies task‑relevant tokens, while an entropy‑based phase identification keeps dense computation in precision phases to avoid performance loss. Experiments on RoboLab, RoboCasa, and real‑world manipulation tasks show consistent speedups—up to 1.33× at 50% pruning—while maintaining task success rates within 1 percentage point of the baseline.

By Baekseung Kim, Junhoo Lee, Sohyeon Hong, Suhyun Jeon, Minkyu Kim, Seungyeon Kim, Donghoon Han, Nojun Kwak
arXiv Computer Vision
Oct 16

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

The paper introduces LICOA, a mobile spatial augmented reality system that treats projected light as an embodied action. LICOA combines a differential‑drive chassis, a 2‑degree‑of‑freedom steerable mirror, and a shared interface that maps synchronized RGB camera inputs to optical actions, enabling the robot to decide how to move, where to project, and at what scale based on language instructions. The authors built a simulation embodiment and a corpus of 150 tasks with 7,500 episodes, then evaluated models trained on this corpus in 480 real‑world closed‑loop trials, achieving a 57.50% success rate and demonstrating the importance of both camera views and coordinated chassis‑mirror control.

By Yuchen Deng, Shunkai Zhou, Ruixuan Zhu, Haibin Ling, Bingyao Huang