arXiv Computer Vision

Unlocking Cross-Attention in Label-Scarce Regimes: A Weakly Supervised Approach for Video Semantic Segmentation

arXiv Computer Vision
Oct 19

Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning

Dual-State Slot Attention (DSSA) is a self‑supervised framework that separates per‑frame appearance from temporally stable identity within each slot, addressing conflicts that cause slot swapping in video object‑centric learning. It introduces a learned recurrent transition for identity updates and competition‑modulated aggregation to suppress weakly matching slots, thereby improving segmentation quality and temporal consistency. Experiments on MOVi‑C, MOVi‑D, and YouTube‑VIS show DSSA outperforms prior methods and enhances downstream object recognition and video dynamics prediction.

By Sieu Tran, Duc Nguyen, Hao Vo, Khoa Vo, Ngan Le
arXiv Computer Vision
Sep 28

StarWM: Self-Supervised Trained Attention Routing for Robust World Models

StarWM introduces a self‑supervised attention routing mechanism that selectively applies reconstruction only to dynamically relevant regions of visual input. By combining a cross‑attention module with a dual‑stream decoder and stop‑gradient barriers, it balances faithful environmental dynamics capture with abstraction of irrelevant content. Experiments on DeepMind Control show that StarWM outperforms both reconstruction‑based and reconstruction‑free baselines, especially under distractor conditions, and preserves state attributes over long‑horizon imagination.

By Zeqiang Zhang, Fabian Wurzberger, Maximilian Otte, Daniel Schmid, Sebastian Gottwald, Arne Peter Raulf, Daniel Alexander Braun
arXiv Computer Vision
Oct 1

DiDA: Video Object Segmentation with Distillation Learning of Deformable Attention

DiDA introduces a lightweight video object segmentation framework that leverages Distillation Learning of Deformable Attention. The method uses deformable attention to adapt key and value positions across frames, enabling object representations that are responsive to spatial and temporal changes. Experiments on DAVIS and YouTube‑VOS benchmarks show state‑of‑the‑art performance and efficient memory usage.

By Quang-Trung Truong, Duc Thanh Nguyen, Binh-Son Hua, Sai-Kit Yeung
arXiv Computer Vision
Oct 28

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

The paper introduces XTrack, a unified framework that repurposes a pretrained video Diffusion Transformer (DiT) for cross-view temporal object correspondence. XTrack incorporates focal cross-view attention alignment, hard negative conditioning, and local context cropping to improve grounding across views and time, and it uses a pseudo‑label curation pipeline to generate dense supervision from sparse Ego‑Exo4D data. The authors demonstrate that XTrack achieves robust temporal consistency and identity preservation in cross‑view tracking tasks.

By Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam, Jaewoo Jung, WonJun Moon, Seungryong Kim
arXiv Computer Vision
Oct 21

TrajTok: Learning Trajectory Tokens enables better Video Understanding

TrajTok introduces an end‑to‑end video tokenizer that replaces traditional patchification with a lightweight, trajectory‑based tokenization process. It performs implicit clustering over space‑time pixels to generate object trajectories in a single forward pass, dynamically adjusting token granularity to semantic complexity while remaining independent of video duration. The resulting TrajViT2 model outperforms existing methods on classification and retrieval benchmarks, and TrajTok can also serve as a probing head (TrajAdapter) or alignment connector (TrajVLM) for vision‑language tasks, notably improving long‑video reasoning.

By Chenhao Zheng, Jieyu Zhang, Jianing Zhang, Weikai Huang, Ashutosh Kumar, Quan Kong, Oncel Tuzel, Chun-Liang Li, Ranjay Krishna
arXiv AI
Jun 12

HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

arXiv:2606. 13289v1 Announce Type: cross Abstract: Holistic visual tokenizers are fundamental to unified multimodal models (UMMs) as they map diverse visual inputs into a unified representation space.

By Guozhen Zhang, Xuerui Qiu, Yutao Cui, Tianhui Song, Changlin Li, Junzhe Li, Tao Huang, Xiao Zhang, Yang Li, Jianbing Wu, Miles Yang, Zhao Zhong, Liefeng Bo, Limin Wang
arXiv Computer Vision
Sep 7

Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval

The paper introduces Intrinsic Temporal Adaptation (ITA) for Partially Relevant Video Retrieval (PRVR), a task that seeks untrimmed videos containing moments relevant to a text query. ITA employs a Backbone-Internal Temporal Adaptation that lets the final visual transformer layers attend to neighboring frames, creating temporally aware embeddings while keeping CLIP frozen. Additionally, an Affinity-Weighted Gradient Propagation technique softly aggregates top‑k frames based on text‑frame affinities to better handle the weakly supervised nature of PRVR, leading to state‑of‑the‑art performance and more accurate frame‑level evidence retrieval.

By Hyun Seok Seong, Woojin Jun, SuBeen Lee, Jae-Pil Heo