Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computer Vision
Oct 9

Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion

The paper introduces SightCAD, a framework for parametric CAD reconstruction that explicitly handles view feasibility constraints in realistic indoor scenes. It learns a view selector to choose a small set of feasible viewpoints for a vision‑language model that generates executable CadQuery code, and jointly trains the selector and the VLM using a reward that depends on the final geometric fidelity. The approach outperforms baseline methods on several benchmarks, achieving higher intersection‑over‑union scores and a lower invalid‑code rate.

By Kartik Bali, Mahish Guru, Yiderigun Borjigin, Alexandra Starostina, Christian J. Cyron, Roland Aydin
arXiv Computer Vision
Oct 9

LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

LIVIN is a new benchmark that evaluates spatial and embodied intelligence using digital twins of 30 diverse lived‑in homes. The replicas preserve real room layouts, furniture configurations, and everyday belongings, created through a human‑in‑the‑loop workflow of instance recognition, architectural reconstruction, and object placement. Four tasks—3D detection, 3D reconstruction, navigation, and loco‑manipulation—demonstrate that current methods struggle with dense object arrangements, occlusions, limited free space, and constrained interaction regions typical of realistic homes.

By Peijun Xu, Chuansen Nie, Yiyang He, Yinuo Bai, Jingyang Liu, Kuixiang Shao, Yuyang Jiao, Kuanhao Xia, Jiayi Zhu, Zitian Yang, Yanqi Zhang, Tianye Tan, Shuwei Di, Junyi Xu, Jingyi Yu, Jiayuan Gu
arXiv Computer Vision
Oct 9

Pumpire: Unified Benchmark for Metric Distance Estimation

Pumpire introduces a unified benchmark for assessing metric point‑pair distance estimation in both image‑ and video‑level 3D foundation models, with or without depth priors. The benchmark uses a large, diverse dataset (pumpire‑6k) of 100 real‑world scenes, each annotated with physically measured point‑pair distances across 64 frames, totaling 6,400 frames. An evaluation protocol is provided that directly measures point‑to‑point distance errors, enabling cross‑setting comparison across 29 baseline 3D models.

By Siyu Chen, Zehan Wang, Jiayang Xu, Yihan Wu, Jialei Wang, Junming Chen, Ziang Zhang, Yutong Ying, Zhou Zhao
arXiv Computer Vision
Oct 9

LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

The paper introduces LEGO, a lifting‑free method for generating egocentric videos from a single exocentric recording. Instead of reconstructing depth and point clouds, it trains a transformer‑based view synthesizer that directly predicts the egocentric view, providing a probabilistic correspondence map that balances structural alignment with texture averaging. This approach outperforms existing explicit pipelines and generalizes across datasets without retraining.

By Suhwan Cho, Yonwoo Choi, Soongjin Kim, Jicheol Park, Taegyu Lim
arXiv Computer Vision
Oct 9

Does Dynamic-Point Filtering Help When Texture Is Scarce? A Controlled Study of ORB-SLAM2 Front-Ends in Synthetic Indoor Scenes

The study evaluates whether dynamic-point filtering improves ORB‑SLAM2 performance in low‑texture indoor scenes by systematically varying texture and scene dynamics in synthetic sequences. Using stereo, RGB‑D, and ground‑truth data, the authors compare ORB‑SLAM2 without filtering, with an optical‑flow/epipolar filter (FLOW), and with a multi‑view depth‑consistency filter (GEOM), measuring trajectory error, tracking completeness, and static‑feature retention. Results show filtering mainly helps in highly dynamic environments; at the lowest texture level it reduces tracking completeness, and ORB‑SLAM2 fails to initialize in static low‑texture scenes, with GEOM discarding more static keypoints than FLOW. "whyItMatters":"The findings challenge the assumption that dynamic‑point filtering always benefits SLAM, highlighting its limited effectiveness in low‑texture settings and guiding future filter design."

By Zekui Xue
arXiv Computer Vision
Oct 9

Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval

The paper introduces the AutoRegressive Retriever (ARR), a multimodal retrieval model that iteratively refines a query embedding by selecting informative items and incorporating their content. ARR alternates between retrieval and query update, using stepwise contrastive supervision and reinforcement learning to optimize item selection and final ranking. Experiments show ARR outperforms baselines on in‑domain and zero‑shot benchmarks, with feedback improving both inference performance and initial query embeddings.

By Jianfei Zhao, Yifan Wang, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan, Yang Luo, Boyuan Pan, Xu Kai, Yao Hu
arXiv Computer Vision
Oct 9

Gaze Estimation for Human-Robot Interaction: Analysis Using the NICO Platform

This paper evaluates current gaze estimation methods in a human‑robot interaction setting using the NICO robotic platform. A new annotated dataset was collected, and four state‑of‑the‑art models were tested, showing angular errors comparable to general benchmarks. In the shared workspace, the best median error was 14.57 cm, highlighting practical limitations and prompting recommendations for integrating gaze estimation into HRI systems.

By Matej Palider, Omar Eldardeer, Viktor Kocur
arXiv Computer Vision
Oct 9

HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving

HENet++ is a unified framework that combines a hybrid image encoding strategy—using a large encoder for short‑term frames and a lightweight encoder for long‑term temporal context—to balance accuracy and efficiency in 3D perception for autonomous driving. It jointly extracts dense background and sparse foreground features, enabling task‑specific representations that reduce cumulative errors and enrich downstream prediction and planning modules. The approach supports diverse 3D feature extraction pipelines, multi‑modal inputs (camera and radar), and achieves state‑of‑the‑art results on nuScenes and NAVSIM benchmarks, including the lowest collision rate on nuScenes planning.

By Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Ming-Hsuan Yang
arXiv Computer Vision
Oct 9

MatLat: Material Latent Space for PBR Texture Generation

The paper introduces MatLat, a material latent space designed for generating high‑quality physically‑based rendering (PBR) textures on 3D meshes. It fine‑tunes a pretrained VAE to incorporate new material channels while preserving latent distribution, and adds a locality‑preserving regularization that aligns cropped latent patches with image regions. Experiments and ablation studies show that these components together improve PBR texture fidelity and outperform prior baselines.

By Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung
arXiv Computer Vision
Oct 9

Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning

The paper introduces Similarity-as-Evidence (SaE), a framework that calibrates overconfident Vision‑Language Models (VLMs) for medical active learning by reinterpreting text‑image similarity vectors as evidence and modeling them with a Dirichlet distribution. SaE’s Similarity Evidence Head (SEH) captures uncertainty through vacuity and dissonance, enabling a dual‑factor acquisition strategy that prioritizes high‑vacuity samples early and high‑dissonance samples later. Experiments on ten public medical imaging datasets demonstrate state‑of‑the‑art macro‑averaged accuracy (82.57%) and improved calibration (NLL = 0.425) under a 20% label budget.

By Zhuofan Xie, Zishan Lin, Jinliang Lin, Jie Qi, Shaohua Hong, Shuo Li
arXiv Computer Vision
Oct 9

Stratified Multi-View Aggregation for Score Distillation

The paper introduces Multi-View Aggregated Score Distillation (MV‑SDI), a training‑free sampler that replaces the single‑view per‑step gradient in score distillation with an average over K views while keeping the diffusion prior frozen. By averaging gradients from multiple views—especially antipodal pairs—MV‑SDI reduces per‑step variance by roughly 1/K, improves angular coverage, and cuts optimization steps while boosting CLIP R‑Precision and CLIP score on the SDI benchmark. The method achieves these gains without retraining or additional multi‑view data and is compatible with existing gradient‑based score‑distillation pipelines.

By Marian Lupascu, Mihai-Sorin Stupariu, Wongi Park, Ionut Mironica
arXiv Computer Vision
Oct 9

ASV3D: Adapting Diffusion-Based Single-View 3D Reconstruction with Extra Imagery

ASV3D is a framework that adapts diffusion-based single-view 3D reconstruction to incorporate an additional image at test time. It offers a zero‑shot adaptation that improves reconstruction quality without retraining, and an optimized adaptation that uses contrastive learning to enhance visual fidelity and cross‑view consistency. The method is applied to two state‑of‑the‑art pipelines, showing consistent gains in accuracy and robustness on benchmark and real‑world datasets.

By Y Huynh, Duc Thanh Nguyen, Thao Minh Le, Mohamed Abdelrazek
arXiv Computer Vision
Oct 9

3D Point World Models: Point Completion Enables More Accurate Dynamics Learning

The paper introduces 3D Point World Models (3DPWM), a task‑agnostic world model that completes partial point clouds before learning action‑conditioned dynamics in 3D space. By operating on fully reconstructed geometry, 3DPWM achieves reliable long‑horizon rollouts (100–300+ steps) and accurate cost evaluation for model‑based planning. Experiments on various robotic embodiments and tabletop manipulation tasks show that 3DPWM supports both open‑loop and closed‑loop planning and enables successful sim‑to‑real transfer.

By Skand Peri, Hung Nguyen, Chanho Kim, Li Fuxin, Stefan Lee
arXiv Statistics ML
Oct 9

Regularized Small Area Estimation with Graph Laplacian Benchmarking priors

The paper introduces a Bayesian framework for small area estimation that jointly handles borrowing information across areas and benchmarking estimates to reliable aggregates via a new family of Benchmarking priors. These priors arise from a benchmark-constrained regularization problem and include a Benchmarking Prior without additional regularization, as well as Single and Multi-View Laplacian Benchmarking Priors that use graph Laplacians built from area similarities based on external covariates. Tailored MCMC algorithms are developed for posterior computation, and the models are evaluated through a data-based simulation and applied to estimate Average Household Size at the municipality level in Colombia in 2025, where graph-based regularization improves performance and Multi-View models yield more precise estimates.

By Jairo F\'uquene-Pati\~no, Brenda Betancourt
arXiv AI
Oct 9

Why machines will still not rule the world

The article argues that artificial general intelligence is mathematically impossible because human intelligence arises from complex systems that cannot be captured by current models. It critiques two common defenses of machine intelligence: the theoretical reliance on universal approximation theorems and the Church‑Turing‑Deutsch principle, and the empirical reliance on benchmark performance. The authors point out flaws in benchmarks, such as data contamination and test construction issues, and conclude that models capable of true cognitive behavior in open‑ended, thermodynamically complex environments will never be achieved.

By Jobst Landgrebe, Barry Smith
arXiv Machine Learning
Oct 9

LadderEdit: Edit-Level Residual Compression for Memory-Efficient Lifelong Editing of LLMs

LadderEdit is a method for memory‑efficient lifelong editing of large language models that compresses each LoRA adapter after acquisition. Each edit is initially stored as a low‑rank sketch and only promoted to higher rank if it fails to satisfy rewrite, generalization, and locality contracts on probe prompts. On benchmarks such as ZsRE, CounterFact, and WikiBigEdit, LadderEdit achieves the same performance as exact LoRA while using 5.2× less memory and handling up to 50,000 sequential edits.

By Xiaobing Yu, Peijie Qiu, Jin Yang, Xuanzhao Dong, Weiwei Ma, Zhaoqi An, Xiaoqi Zhao, Xiaofeng Liu
arXiv AI
Oct 9

How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices

The paper surveys how automated research systems—used for literature synthesis, ideation, workflows, writing, peer review, and end‑to‑end research—are evaluated. It compares task construction, evidence sources, evaluators, and scoring methods across six target areas, revealing that output checks, process checks, and human studies together provide a fuller picture. The authors identify gaps in diagnostic designs and offer reporting and audit recommendations to guide future evaluation efforts.

By Liulei Zhang, Dejing Zhou, Chuyue Huang, Guanhua Chen, Yutong Yao, Lidia S. Chao, Chi Man Vong, Derek F. Wong
arXiv AI
Oct 9

DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration

DuplexAgent-RSI is a full‑duplex voice‑agent collaboration system that coordinates continuous listening and speaking with asynchronous delegation to reasoning and coding agents. Its harness is modular, consisting of six editable components, and a closed‑loop improvement process (Duplex‑Harness‑RSI) revises these modules based on interaction traces and failure diagnostics. Experiments show that this approach yields stronger spoken‑knowledge and executable‑tool performance than comparable delegated systems while preserving robust interruption response.

By Yingda Shen, Yuxiang Wang, Kunyu Feng, Qinke Ni, Jiaqi Li, Minghao Hsu, Junan Zhang, Dekun Chen, Yutong Bian, Zhizheng Wu
arXiv Machine Learning
Oct 9

LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression

LAIR-Net is a Leaky Alignment-Impulse Residual Network designed for tabular regression that incorporates a shallow learned anchor into each hidden state via a leaky residual transition. The authors provide a depth‑uniform bound on input‑perturbation sensitivity and show through controlled simulations that performance gains over randomized baselines stem from the anchor rather than recursion or added capacity. On 23 benchmark datasets, LAIR‑Net achieves the best average rank among eight randomized networks and twelve conventional models, with its success linked to the presence of nonlinear target structure and manageable noise levels.

By Rahul Goswami, Aryan Bhambu, Bittu Karmakar
arXiv AI
Oct 9

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

Guard models are used to prevent LLM agents from performing forbidden actions, but this alone does not guarantee safety. The paper shows that many agent trajectories contain unfulfilled safety‑critical obligations—actions that should have been performed—highlighting a new source of risk. To address this, the authors introduce ObligationBench, a benchmark of 240 expert‑validated trajectories, and develop ObligationGuard, a model that improves obligation detection recall and exact‑match rates over existing models.

By Youwei Feng, Yitong Zhang, Yuetong Liu, Jia Li