Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
Oct 23

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning

The paper introduces VANE, a self‑supervised visual pretraining framework that learns from next‑frame prediction in embedding space. It uses a causal encoder to summarize past observations, a state predictor to infer the representation of the entire next frame, and a lightweight generative decoder to reconstruct fine‑grained spatial detail. By applying the same objective to both videos and still images through a local‑to‑global crop sequence, VANE achieves state‑of‑the‑art performance on anticipation, recognition, and dense perception benchmarks, notably outperforming V‑JEPA on EPIC‑KITCHENS‑100 action anticipation.

By Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao
arXiv Computer Vision
Oct 23

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

The paper introduces CAVC, a training‑free, modality‑asymmetric caching framework that speeds up diffusion‑based audio‑visual generation by selectively caching audio denoiser outputs while continuing full video refinement. By detecting when audio denoiser outputs become cache‑tolerant, CAVC bypasses audio denoising and cross‑modal attention in the middle of the sampling trajectory, then restores a few joint steps at the end to correct drift and improve synchronization. Experiments on public benchmarks show 1.10–1.35× wall‑clock acceleration, better visual dynamics and temporal statistics, and competitive audio fidelity and alignment.

By Lifan Jiang, Yibo Zhao, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
arXiv Computer Vision
Oct 23

Direct Forcing: A Single-Stage Training Recipe for Few-Step Autoregressive Text-to-Video Generation

Direct Forcing is a single‑stage training framework for few‑step autoregressive text‑to‑video generation that eliminates the need for separate causal adaptation and distribution matching stages. It uses reference videos both as paired examples for causal continuation alignment and as a target distribution for rollout distribution matching, with dynamic gradient balancing to adjust their contributions during training. Starting from a bidirectional Wan2.1‑1.3B model, Direct Forcing trains a four‑step generator in only 250 updates, achieving state‑of‑the‑art VBench scores without additional initialization or diffusion‑score supervision.

By Haojun Xu, Jie Huang, Xin Lu, Mingchen Zhong, Zihao Fan, Linjiang Huang, Si Liu
arXiv Computer Vision
Oct 23

ResGVC: Residual Coding for High-fidelity Generative Video Compression

ResGVC introduces a residual coding framework for generative video compression that transmits residual signals to guide denoising directly in pixel space. The method encodes original frames with a learned video codec, decomposes the reconstruction error into residual segments at each denoising step, and compresses these segments using a residual codec powered by a diffusion transformer entropy model. Compared to prior state‑of‑the‑art methods, ResGVC achieves significant BD‑rate reductions—63.7% for LPIPS and 83.8% for PSNR—while delivering superior visual quality.

By Ding Ding, Daowen Li, Ying Chen, Kai Li
arXiv Computer Vision
Oct 23

I$^2$-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis

I$^2$-KAN is a new multimodal sentiment analysis framework that separates intra‑modal and pairwise inter‑modal information into distinct pathways. Each pathway is refined by a Task‑Decoupled Linear‑Gated Denoising Bottleneck and fused using a structured Kolmogorov‑Arnold Network, enabling explicit pathway associations and multi‑level interpretation. Experiments on CMU‑MOSI, CMU‑MOSEI, and CH‑SIMS v2 show strong performance and reveal that a modality’s standalone predictive power does not always reflect its interaction‑conditioned utility.

By Xiaoting Xie, Feng Qiao, Xianwei Zhang, Hailong Chen, Kaiwen Zheng, Joemon M. Jose, Xuri Ge, Yujun Li
arXiv Computer Vision
Oct 23

WildDepth-v2: A Synchronized RGB--LiDAR--Radar Dataset and Benchmark for Metric Depth, Motion, and Tracking of Wildlife

WildDepth‑v2 expands the original RGB‑LiDAR WildDepth dataset by adding 37 RGB‑LiDAR‑4D‑radar sequences, a frozen hidden test set for RGB‑only depth estimation and RGB‑plus‑sparse‑LiDAR depth completion, and metric 3D trajectories derived from 2D tracks using LiDAR, radar, or monocular depth. The dataset demonstrates that object‑level radar calibration improves monocular depth accuracy (AbsRel 0.120 vs. 0.198) and that once‑per‑track radar scaling reduces 3D position error four‑fold, matching the performance of LiDAR calibration. Additionally, transferring radar Doppler onto LiDAR yields denser motion masks, lowers LiDAR accumulation error, and provides radial speed estimates. whyItMatters":"The dataset supplies the first imaging‑radar wildlife recordings and a benchmark that enables metric depth, motion, and tracking research in ecological contexts, filling a gap where monocular video lacks scale and existing wildlife datasets lack metric ground truth."

By Muhammad Aamir, Sangyun Shin, Naoya Muramatsu, Matthew Wijers, Amir Patel, Andrew Loveridge, Andrew Markham
arXiv Computer Vision
Oct 23

A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models

The paper introduces RetinaGroundBench, a new benchmark for visual grounding of retinal OCT images, featuring 815 B-scans annotated with 4,537 bounding boxes for 13 biomarkers across 8 diseases. It evaluates 27 multimodal large language models (MLLMs) and finds that visual grounding is largely unsolved, with the best model achieving only a macro F1 of 23.7%. The authors also present RetGrounder, an 8‑B parameter MLLM fine‑tuned for OCT, which raises grounding F1 to 33.9%, a 43% relative improvement.

By Ronald Fecso, Jos\'e Morano, Botond Fazekas, Paul Widmann-Sedlnitzky, Dorota Himel, Laetita Hinterhuber, Virginia Mares, Andreas Pollreisz, Hrvoje Bogunovi\'c
arXiv Computer Vision
Oct 23

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

The paper introduces MoRSe, a method that segments only the regions that actually move between two frames, rather than labeling entire objects as moving. Unlike prior approaches that rely on instance-level proposals and often misattribute motion to whole objects, MoRSe jointly interprets two frames with a single RGB encoder, eliminating the need for optical flow or separate mask generators. The authors also release MoRSe-5K, a benchmark of 42 sequences designed to test the model’s ability to differentiate moving and stationary regions under various camera and scene motion conditions, and demonstrate that MoRSe outperforms existing moving object segmentation and dense 4D reconstruction methods with fewer false positives.

By Wenhua Jin, Tiancheng Chen, Sheng Tang, Jiahao Li, Yupeng Liu, Weiqi Zhang, Juntong Fang, Zesong Li
arXiv Computer Vision
Oct 23

GLOW-FM: Efficient Generative Low-Light Image Enhancement with One-Step Latent Flow Matching

GLOW-FM is a one‑step latent flow‑matching framework designed for efficient generative low‑light image enhancement (LLIE). It operates in the latent space of a frozen Tiny‑VAE, using Terminal Velocity Matching (TVM) to achieve high‑quality enhancement with a single function evaluation. Experiments on real‑world benchmarks show competitive image quality, strong cross‑dataset generalization, and low computational cost (34 GFLOPs for the small variant).

By Ariel Lapid, Yuval Haitman, Reuven Peretz, Arnon Netzer, Idit Diamant
arXiv Computer Vision
Oct 23

MVLoMa: Multi-View Local Feature Matching with Any Geometry Foundation Model

MVLoMa is a sparse star‑graph matcher that extends the LoMa framework from image pairs to multi‑view settings. It uses a frozen multi‑view geometry model to jointly describe all images in a star, injecting cross‑view geometric context into local descriptors and matching the centre image to all neighbours in a single forward pass. On several datasets, MVLoMa achieves the best macro‑average relative‑pose AUC and is more than twice as fast as sequential pairwise evaluation, while supporting multiple frozen geometry encoders and demonstrating the benefits of geometry‑aware descriptors, visibility filtering, and joint inference.

By Xinyang Lu, Shengzhe Wei, Sheng Tang, Weiqi Zhang, Juntong Fang, Zesong Li
arXiv Computer Vision
Oct 23

MixPT: Point Transformer with Interleaved Window and Global Attention

MixPT is a point transformer that combines window and global attention blocks in an interleaved schedule, using a fixed serialization pattern for window blocks. It applies global attention to spatially pooled tokens for large scenes, feeding residual updates back to the original-resolution stream. Experiments on indoor and outdoor benchmarks show MixPT achieves competitive accuracy with lower latency than existing window- or global-attention baselines, and can scale by repeating the interleaved pattern.

By Minkyun Seo, Hyungtae Lim, Jaesik Park
arXiv Computer Vision
Oct 23

F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

F2G-Pose is a real‑time, single‑pass framework for category‑level RGB‑D object pose estimation that predicts 3D rotation, translation, metric size, and a completed camera‑frame shape from a single RGB‑D observation. It lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and uses a Mixture‑of‑Experts Transformer to integrate object‑level structure. Trained only on pose‑annotated synthetic SOPE data and leveraging a frozen RADIO backbone, F2G‑Pose achieves state‑of‑the‑art accuracy on SOPE and ROPE while running at 28 FPS for pose‑only inference and demonstrates strong geometry completion on HouseCat6D and real‑object transfer on HANDAL.

By Jinyu Zhang, Haitao Lin, Jiashu Hou, Xiangyang Xue, Yanwei Fu
arXiv Computer Vision
Oct 23

UniVOE: A Removal-Driven Framework for Interaction-Aware Universal Video Object Editing

UniVOE is a removal‑driven framework that trains both video object removal and injection models. It first builds an effect‑aware remover by leveraging video inpainting priors and fine‑tuning on object‑present/object‑absent pairs, then uses the remover to generate large‑scale paired data for training an injection model that restores objects with their dynamics and interaction effects. A reward‑based post‑training step aligns injection results with human preferences, and experiments show UniVOE outperforms existing open‑source and commercial systems, especially in challenging interaction‑effect scenarios.

By Shuzhou Yang, Lanqing Guo, Lingen Li, Weiqi Li, Zhao Zhong, Liefeng Bo, Jian Zhang
arXiv Computer Vision
Oct 23

Gate3DGen: A Pre-processing Model that Selectively Purifies Real-World Images for 3D Generation

Gate3DGen is a selective pre‑processing model that improves single‑image 3D generation by purifying real‑world images with uneven illumination, blur, or occlusions. It uses a degradation‑aware router to bypass clean inputs and a 3D‑oriented one‑step purifier that preserves object identity, contours, and textures. The method is trained with a synthetic data engine and demonstrates significant improvements on synthetic benchmarks and real photographs, reducing mesh distance by 61.5% and receiving positive human and VLM evaluations.

By Xiaolin Li, Zihao Huang, Fanpeng Meng, Chen Yang, Guanjun Wu, Kai Zhang, Jiemin Fang, Qi Tian
arXiv Computer Vision
Oct 23

Reading, Writing, Sharing, and Interacting with the World through Video

The paper surveys recent progress in video foundation and world models, framing both as ways to relate video observations to an evolving latent world. It categorizes the literature into four operations—reading, writing, sharing, and interacting—highlighting how each addresses different aspects of understanding, generating, integrating, and updating world states from video. The authors trace a shift from clip‑level recognition toward persistent, controllable, and physically valid world representations, and outline remaining challenges.

By Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei
arXiv Computer Vision
Oct 23

Bound-Conditioned Latent Inference for Progressive Image Compression

Bound-Conditioned Latent Inference (BLI) redefines progressive trit-plane coding as sequential latent inference under shrinking interval constraints, using decoded trits to set lower and upper bounds for each latent element. BLI jointly refines Gaussian parameters and latent estimates with hyperprior information, enabling tighter constraints and flexible refinement schedules. Experiments on Kodak, CLIC, and JPEG-AI demonstrate state‑of‑the‑art rate‑distortion performance, reducing BD‑rate by 17.78% over DPICT and 7.96% over CTC while using far fewer parameters and lower latency.

By Jaeseok Jang, Seungmin Jeon, Kwang Pyo Choi, Chang-Su Kim
arXiv Computer Vision
Oct 23

Hybrid-PCGC: Hybrid Point Cloud Geometry Compression Bridging Generalization-based Model and Implicit Neural Representation

Hybrid-PCGC is a point cloud geometry compression method that blends learned generalization with implicit neural representations. It treats compression as a joint geometry-and-model bitrate allocation problem, optimizing instance‑specific weight offsets as transmitted variables. The approach separates dataset‑agnostic prior modeling from instance‑specific adaptation using a Frozen General Module, a Dual Trainable Adaptive Module, an Adaptive Scope Controller, and a Supervised Offset Compression Module, achieving significant bitrate reductions over existing methods.

By Wenjie Huang, Qi Yang, Shuting Xia, He Huang, Zhu Li, Yiling Xu
arXiv Computer Vision
Oct 23

SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA

SpatialMem is a memory-based system that fuses 3D geometry, semantics, and language into a single queryable representation. Starting from casually recorded egocentric RGB video, it reconstructs metrically scaled interior scenes, identifies structural 3D anchors (walls, doors, windows), and links open‑vocabulary object nodes to evidence patches, visual embeddings, and two‑layer descriptions at 3D coordinates, enabling spatial‑relation reasoning, language‑directed retrieval, and QA without special sensors. The system is evaluated at scale on public benchmarks, achieving competitive scores on VSI‑Bench, OpenEQA, and ScanNet, while maintaining a compact index of about 14 KB per scene.

By Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen
arXiv Computer Vision
Oct 23

ORSIFlow: Saliency-Guided Rectified Flow for Optical Remote Sensing Salient Object Detection

ORSIFlow is a saliency‑guided rectified flow framework for optical remote sensing salient object detection that reframes the task as deterministic latent flow generation. It uses a frozen variational autoencoder to produce saliency masks in a compact latent space, allowing efficient inference with few steps. The method incorporates a Salient Feature Discriminator for global semantic discrimination and a Salient Feature Calibrator for precise boundary refinement, achieving state‑of‑the‑art performance on multiple public benchmarks.

By Haojing Chen, Zhihang Liu, Yutong Li, Tao Tan, Haoyu Bian, Qiuju Ma
arXiv Computer Vision
Oct 23

Fast and Compact Graph Cuts for the Boykov-Kolmogorov Algorithm

The paper revisits the Boykov‑Kolmogorov (BK) algorithm for computing minimum s‑t cuts in graphs, improving its theoretical time complexity to O(mn|C|). It introduces a faster variant, the fast BK (fBK) algorithm, with complexity O(m|C|), and proposes a compact graph representation enabling the algorithms to handle graphs with up to 10^9 vertices and 10^10 edges on a 128 GB machine. Experiments show that the compact BK implementation (cBK) outperforms other BK variants, underscoring the value of memory‑efficient designs.

By Christian M{\o}ller Mikkelstrup, Anders Bjorholm Dahl, Philip Bille, Vedrana Andersen Dahl, Inge Li G{\o}rtz