Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language
Oct 23

UniGuard: Multi-View Guardrail Learning for Cross-Interface Safety Generalization

UniGuard introduces a training framework that learns interface‑invariant safety representations by leveraging multiple semantically aligned views of the same safety case. The authors created the UniGuard‑FT dataset, transforming 1,000 public safety cases into prompt, response, multi‑turn conversation, agent‑trace, and multi‑agent‑trace formats, and trained a 0.6B model using classification, supervised contrastive learning, and asymmetric confidence calibration losses. On unseen‑interface benchmarks (ATBench, PolicyGuardBench, DynaBench), UniGuard‑0.6B outperforms its pretrained guardrail by an average of 45.12 F1 points and reduces expected calibration error by 35.41 points, while remaining competitive with much larger guardrails such as Claude‑Sonnet‑4.5 and GPT‑5.2.

By Viet-Thanh Pham, Thuy-Trang Vu, Junae Kim, Rollin Omari, Dinh Phung, Gholamreza Haffari, Lizhen Qu
arXiv Computation and Language
Oct 23

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

ReflectPO introduces a reinforcement learning framework that enables large language model agents to recover from errors during long-horizon tasks involving external tools. The method uses Reflection-Guided Sampling to generate recovery hints from branch differences and repeated visits, and Tree-Structured Credit Assignment to allocate action-level credit based on parent–child reward differences and recovery gains. Experiments on two tool-use benchmarks demonstrate that ReflectPO outperforms existing baselines and improves agents’ ability to detect and correct deviations, thereby advancing autonomous error recovery in complex real-world environments.

By Rumei Li, Xunliang Cai
arXiv Computation and Language
Oct 23

The Imperfective Paradox in Large Language Models

The paper examines whether large language models truly understand the compositional semantics of events, focusing on the Imperfective Paradox where past progressive implies event completion for activities but not for accomplishments. It introduces the ImperfectiveNLI diagnostic dataset to test this distinction across semantic classes. Experiments reveal a Teleological Bias in open-weight models, where they hallucinate completion for goal-oriented events and struggle with atelic verbs, suggesting that these models function more as predictive narrative engines than logical reasoners.

By Bolei Ma, Yusuke Miyao
arXiv Computation and Language
Oct 23

TORUS: A Unified Closed-Loop Benchmark for Unified Audio Models

The paper introduces TORUS, a closed‑loop benchmark designed to evaluate unified audio models on their ability to understand the audio they generate or edit. TORUS contains 1,053 questions across 21 tasks and 8 audio domain settings, covering all three audio types. The authors evaluate both unified models and cascades of specialized models, finding that the best unified models answer only about 52.4% of questions about their own outputs, while GPT‑Audio achieves 53.6%.

By Aryan Vijay Bhosale, Harshit Rajgarhia, Abhishek Mukherji, Dinesh Manocha
arXiv Computer Vision
Oct 23

R4DGS: Referring Segmentation in 4D Gaussian Splatting

The paper introduces R4DGS, a method for referring segmentation in dynamic 4D Gaussian splatting scenes. It presents the R4D-Bench-QA benchmark with structured query annotations and a unified framework called ReferGaussian that converts dynamic Gaussians into language-addressable entities for training‑free inference. Experiments show improved temporal target selection and competitive spatiotemporal grounding without scene‑specific semantic training.

By Bangpu Chen, Yaxuan Li, Shirui Peng, Xiangtian Si, Liuxin Chu, Xitong Cao, Hongbo Jin, Jiayu Ding
arXiv Computer Vision
Oct 23

AVAtt: A Free-Viewing Eye-Tracking Dataset and Benchmark for Visual Attention on Paintings

AVAtt is a free‑viewing eye‑tracking dataset comprising Images{} paintings from sixteen art movements and five centuries, with Scanpaths{} scanpaths and Fixations{} fixations collected from Participants{} participants. The dataset provides individual scanpaths, fixation timing, eye‑screen distance, fixation and saliency maps, temporal saliency videos, and rich art metadata. A benchmark of classic and deep saliency models shows significant room for improvement, and 44 scanpath‑similarity measures—including a new language‑free image‑encoder family—are ranked.

By Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Alessandro Bruno
arXiv Computer Vision
Oct 23

Evaluation of Front-End Feature Matching for Lunar Visual SLAM

The study evaluates eleven feature detector–matcher combinations for lunar visual SLAM using synthetic, lunar-testbed, and terrestrial-analogue datasets. SuperPoint pairings outperform SIFT with ratio-tested FLANN in a rotation-only metric on lunar testbeds, but SIFT excels when rotation and translation are jointly scored on synthetic data. In stereo VO, SuperPoint maintains tracking across all datasets, while classical detectors are more sensitive to lighting conditions, and the fastest learned combination is SuperPoint with LightGlue.

By Guillem Casadesus Vila, Grace Gao
arXiv Computer Vision
Oct 23

One Evaluation Is Enough: Inversion-Free Image Editing with Trajectory Correction

The paper introduces Endpoint‑Calibrated Transport (ECT), a closed‑form, training‑free image editor that performs inversion‑free editing with a single temporal field evaluation. ECT leverages a duality in variance‑preserving one‑step models to compute a local angular semantic tangent and a clean endpoint displacement from one source‑target query at a shared noisy state. By integrating the tangent over the remaining VP angle and correcting the extrapolation with the endpoint displacement, ECT requires only one noise level, two text conditions in one denoiser batch, and a single latent update, achieving stable quality gains in very short editing time.

By Lifan Jiang, Taiyang Chen, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
arXiv Computer Vision
Oct 23

GraphCAD: Geometry-Aware Diffusion on Edge-Adjacency Graphs for CAD Generation

GraphCAD introduces a geometry-aware generative framework for Computer-Aided Design (CAD) that models CAD construction using an Edge-Adjacency Graph (EAG), a novel graph-based representation. The method employs a hybrid graph diffusion process to jointly capture discrete topological connectivity and continuous spatial geometry, and incorporates a geometry-aware denoising mechanism that maps node attributes to SE(3)-equivariant encoded point sets. Experiments on the newly created EAGBench dataset demonstrate that GraphCAD can generate complex CAD models with more than 50 faces, achieving state‑of‑the‑art geometric fidelity and distributional alignment compared to sequence‑based baselines.

By Wenhao Zheng, Xianggen Liu, Li Yue, Jiancheng Shi, Jiancheng Lv
arXiv Computer Vision
Oct 23

Where Do Long Captions Fail? Position-Aware Diagnosis and Reinforcement Learning for Detailed Image Captioning

The paper introduces a position‑aware rubric that breaks down long image captions into atomic visual claims, allowing verification of generated text against these claims and revealing distinct failure modes: unsupported specificity in the middle and vague language at the end. Using this diagnosis, the authors propose PAGER, a reinforcement learning objective that provides separate rewards for middle evidence, tail continuation, claim coverage, and length control. Experiments on detailed‑caption benchmarks show that PAGER improves position‑wise grounding and pairwise caption preference while maintaining overall caption quality.

By Yuanze Hu, Zhichao Yang, Junwei Jing, Xin Yang, Wenxuan Liu, Tinghai Zhang, Jia Fu, Hongzhi Zhang, Xiangyu Wu, Ruiming Tang, Tingting Gao, Han Li
arXiv Computer Vision
Oct 23

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

HiSTEVE 1.0 is a hierarchical VLA agent designed to operate in open‑world games like Minecraft, combining long‑horizon autonomy with native interface control. The system uses a capability‑aware harness with structured memory, a VLM planner, and a policy‑coupled task completion detector to orchestrate subtasks, while a single VLA handles vision‑only perception and native keyboard‑mouse actions. HiSTEVE achieves state‑of‑the‑art performance on atomic tasks and successfully completes long‑horizon tasks such as crafting an iron pickaxe from scratch.

By Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai
arXiv Computer Vision
Oct 23

WeVisOne-VL: Unified Visual Perception from Objects to Pixels

WeVisOne‑VL is a new vision‑language model that unifies object‑level and pixel‑level perception through structured next‑token prediction. It supports a wide range of visual tasks—object detection, referring localization, counting, OCR, segmentation, and matting—allowing localization at any user‑specified granularity from points and boxes to masks and alpha mattes. The architecture incorporates a lightweight pixel decoder via a special token, and the model is trained with a three‑stage curriculum followed by mixed‑task reinforcement learning, achieving leading performance on over 30 benchmarks while robustly rejecting object‑absent queries.

By Tianming Liang, Junyi Pan, Yuxi Li, Canmiao Fu, Zhipeng Huang, Chong Sun, Chen Li, Jing Lyu
arXiv Computer Vision
Oct 23

CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

CellTrack-Bench is a new benchmark designed to evaluate Vision‑Language Models (VLMs) on microscopic cell tracking tasks. It compiles 1,537 problems from 15 standard cell‑tracking datasets across four imaging modalities, totaling 9,041 cell‑linking decisions, and includes a hard subset that highlights shared failures among specialized trackers. The benchmark was used to assess 9 closed‑source and 13 open‑weight VLMs, compare them to specialized trackers, and analyze their failure modes, with all code and data available on GitHub.

By Ilan Theodoro, Jord\~ao Bragantini, Lo\"ic A. Royer
arXiv Computer Vision
Oct 23

OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields

OmniRay is a feed‑forward model for monocular 3D reconstruction that works without assuming a pinhole camera. It predicts a ray direction and a scale‑free radial distance for each pixel using a composite Ray Module consisting of a monotone radial base, a spherical‑harmonic residual, and a free‑form residual. Experiments across various camera types show that OmniRay achieves state‑of‑the‑art performance in camera‑agnostic monocular 3D reconstruction.

By Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa, Anand Bhattad
Hugging Face Trending Papers
Oct 22

Causal Chain of Thought: Mitigating Attention Leakage via Context Isolation

CausalCoT is an inference framework that improves the causal transparency of chain-of-thought prompting by structuring reasoning as a directed acyclic graph of atomic sub-questions. Each node is evaluated in an isolated context that includes only its direct causal parents, preventing attention leakage to unrelated content. This approach maintains competitive predictive performance while significantly enhancing explanation faithfulness and enabling predictable test-time interventions.

Hugging Face Trending Papers
Oct 22

F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

F2G-Pose is a real‑time, single‑pass framework for category‑level RGB‑D object pose estimation that predicts pose, metric size, and camera‑frame completed shape from a single RGB‑D observation. It lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and uses a Mixture‑of‑Experts Transformer to integrate object‑level structure. Trained only on pose‑annotated synthetic SOPE data with a frozen RADIO backbone, it achieves state‑of‑the‑art accuracy on SOPE and ROPE while running at 28 FPS for pose‑only inference and demonstrates strong geometry on HouseCat6D and real‑object transfer on HANDAL.

Hugging Face Trending Papers
Oct 22

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

The study presents an evidence‑aware evaluation protocol for a retrieval‑augmented generation system focused on breast lesion management literature. Using an 18‑document corpus and 18 queries, the authors compared dense embeddings, BM25, and cross‑encoder reranking across various chunk sizes and overlaps, finding that a hybrid retriever with 384‑token chunks and 20% overlap achieved top‑three evidence retrieval for 17 of 18 queries and high recall and precision metrics. Ground‑truth curation expanded evidence spans and accepted chunk targets, demonstrating that annotation coverage significantly affects retrieval performance.

Hugging Face Trending Papers
Oct 22

VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act

VERA is an open‑source, self‑hosted dashboard designed to streamline responsible AI assessments for large language model deployments in regulated environments, such as a European bank. It interprets the EU AI Act into six principle groups and eighteen technical requirements, scoring twelve that current benchmarks can measure while also tracking energy consumption for environmental impact. A within‑subject study showed that using VERA cut assessment time from 56 to 30 seconds and received a high usefulness rating of 4.36 out of 5.