arXiv Computer Vision
Oct 21

TrajTok: Learning Trajectory Tokens enables better Video Understanding

TrajTok introduces an end‑to‑end video tokenizer that replaces traditional patchification with a lightweight, trajectory‑based tokenization process. It performs implicit clustering over space‑time pixels to generate object trajectories in a single forward pass, dynamically adjusting token granularity to semantic complexity while remaining independent of video duration. The resulting TrajViT2 model outperforms existing methods on classification and retrieval benchmarks, and TrajTok can also serve as a probing head (TrajAdapter) or alignment connector (TrajVLM) for vision‑language tasks, notably improving long‑video reasoning.

By Chenhao Zheng, Jieyu Zhang, Jianing Zhang, Weikai Huang, Ashutosh Kumar, Quan Kong, Oncel Tuzel, Chun-Liang Li, Ranjay Krishna
arXiv AI
Oct 16

Memory as Weights: Internalizing Long-Term History for Streaming Videos

The paper introduces MaLoW, a streaming memory framework that internalizes long‑term video history into evolving lightweight parameter updates using a recurrent HyperNetwork and LoRA weights. MaLoW keeps current observations as input while carrying historical context forward without re‑visiting the full video stream. The authors also present MaLoW‑Instruct, a large streaming video dataset, and show that MaLoW outperforms prior memory methods on several benchmarks and improves action success in a vision‑language‑action setting.

By Zhuoming Liu, Jinhong Lin, Lin Zhang, Yiwu Zhong, Yin Li
arXiv Computer Vision
Nov 5

Towards Visual Query Segmentation in the Wild

The paper introduces Visual Query Segmentation (VQS), a new task that segments all pixel-level occurrences of a queried object in untrimmed videos. It presents VQS-4K, a large benchmark with 4,111 videos, 1.3 million frames, and 222 object categories, each paired with a visual query and annotated masklets. The authors also propose VQ‑SAM, an extension of SAM 2 that uses target‑specific and background cues to improve VQS performance, achieving state‑of‑the‑art results on VQS‑4K.

By Bing Fan, Minghao Li, Hanzhi Zhang, Shaohua Dong, Naga Prudhvi Mareedu, Xiaoqiong Liu, Weishi Shi, Yunhe Feng, Yan Huang, Heng Fan
arXiv AI
Aug 28

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

LeVJEPA is a video encoder that eliminates the need for architectural asymmetries, exponential-moving-average target encoders, stop-gradients, and capacity-limited predictors used in prior self‑supervised methods. It trains a single encoder with an invariance loss over global and local views, regularized by SIGReg to prevent collapse, and achieves strong performance with far less pretraining compute. The approach also allows block‑causal attention, making temporal ordering a property of the encoder itself, and matches or surpasses state‑of‑the‑art baselines on both appearance‑centric and motion‑centric benchmarks.

By Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner
arXiv Computer Vision
Nov 6

Training-Free Predictive Spatio-Temporal Token Pruning for Few-Step Autoregressive Video Diffusion

The paper introduces PreSTP, a training‑free method for speeding up few‑step autoregressive video diffusion models by pruning redundant tokens. It identifies two redundancy patterns—cross‑frame similarity from static content and cross‑step convergence during denoising—and uses early clean estimates from the model’s own trajectory to build a token‑wise redundancy mask. PreSTP achieves significant speedups (up to 3.20× for 5‑second videos and 1.96× for 30‑second videos) while maintaining or improving fidelity compared to existing caching baselines.

By Jia Li, Xurui Peng, Xing Wang, Hayden Kwok-Hay So