arXiv AI

Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding

Pocket-STVG (P-STVG) is a lightweight cascade architecture for Spatio-Temporal Video Grounding that combines efficient pre‑trained components: a temporal‑aware video encoder based on MobileViCLIP, a spatial encoder‑decoder from MDETR, and a shared aligned text encoder. Temporal localization is achieved with a lightweight 1D U‑Net or a simple thresholding strategy, allowing the model to work in both weakly supervised and zero‑shot settings. With fewer than 90 M parameters, P-STVG matches or surpasses prior weakly supervised and zero‑shot methods while offering a more memory‑ and compute‑efficient pipeline for large‑scale video collections.

arXiv Computer Vision

Aug 31

Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding

The paper introduces Parallel Tube Decoding (PTD), a generative approach for spatio‑temporal video grounding that splits the task into a temporal block and simultaneous time‑conditioned spatial blocks, eliminating token‑level and trajectory‑level dependencies. PTD uses Decoupled Block Attention to allow parallel spatial generation while maintaining shared video‑query context, and incorporates localization‑aware policy optimization for temporal boundaries and spatial geometry. Experiments on VidSTG show PTD cuts tube completion latency by 79× and boosts spatial decoding throughput by 92× compared to autoregressive decoding, while improving grounding accuracy and performing well on related tasks such as temporal grounding, VideoQA, and referring video object tracking.

By Hanoona Rasheed, Haania Siddiqui, Ming-Hsuan Yang, Fahad Shahbaz Khan, Salman Khan

llms efficiency multimodal

arXiv AI

Jul 16

ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding

arXiv:2607. 13421v1 Announce Type: cross Abstract: Spatio-Temporal Video Grounding (STVG) aims to retrieve the visual trajectory of a specific object from a video stream as described by a natural language expression.

By Kai Chen, Ming Dai, Wenxuan Cheng, Wankou Yang

multimodal benchmarks

arXiv AI

Jul 28

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

arXiv:2607. 24570v1 Announce Type: cross Abstract: Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video.

By Jiameng Zhang, Srikanth Madikeri

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 6

ZoomV: Temporal Zoom-in for Efficient Long Video Understanding

arXiv:2504. 01407v3 Announce Type: replace-cross Abstract: Long video understanding poses a fundamental challenge for large video-language models (LVLMs) due to the overwhelming number of frames and the risk of losing essential context through naive downsampling.

By Yuan Zhang, Junwen Pan, Rui Zhang, Xin Wan, Qizhe Zhang, Ming Lu, Qi She, Shanghang Zhang

llms agents benchmarks

arXiv AI

Jul 29

Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding

arXiv:2607. 24794v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) demonstrate superior generalization in fundamental video tasks, restricted context windows limit their long video understanding.

By Linghao Meng, Qiankun Li, Junyuan Mao, Pujin Liao, Zhicheng He, Enbo Zhang, Kun Wang, Yang Liu, Huazhu Fu, Yueming Jin

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 22

RefAM: Attention Magnets for Zero-Shot Referral Segmentation

arXiv:2509.22650v3 Announce Type: replace Abstract: Most existing approaches to referring segmentation achieve strong performance only through fine-tuning or by composing multiple pre-trained models,...

By Anna Kukleva, Enis Simsar, Alessio Tonioni, Muhammad Ferjad Naeem, Federico Tombari, Jan Eric Lenssen, Bernt Schiele

llms diffusion computer-vision fine-tuning multimodal benchmarks

arXiv Computer Vision

Aug 25

OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation

arXiv:2608.14835v2 Announce Type: replace Abstract: Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as $\langle$subject, predicate, object$\rangle$ triplets, and underp...

By John Helsby, Yi Yang, Bodo Rosenhahn, Michael Ying Yang

llms nlp efficiency benchmarks safety

Hugging Face Trending Papers

Sep 9

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a te...

llms nlp

arXiv AI

Jul 14

Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

arXiv:2607. 09876v1 Announce Type: cross Abstract: Automatically retrieving videos from large camera-trap datasets remains challenging.

By Valentin Gabeff, Baptiste Maquignaz, Jennifer Shan, Sepideh Mamooler, Gencer Sumbul, Blair Costelloe, Devis Tuia, Alexander Mathis

llms agents diffusion computer-vision multimodal benchmarks safety

arXiv AI

Jul 14

Evidence-Backed Video Question Answering

arXiv:2607. 11862v1 Announce Type: cross Abstract: Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding.

By Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong, Silvio Savarese, Chen Sun, Juan Carlos Niebles

llms computer-vision nlp fine-tuning benchmarks

arXiv Computer Vision

Sep 3

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

TAME is a CLIP‑based framework for Text‑Video Retrieval that incorporates temporal modeling through three key innovations: sparse Mixture‑of‑Experts layers with frame‑consistent routing, Frame‑Temporal tokens that aggregate cross‑frame information, and a Cross‑Temporal Interaction and Aggregation module for refining frame‑wise similarities. These components enable the model to capture both local visual patterns and long‑range temporal dependencies, leading to consistent performance gains over CLIP‑based baselines on multiple TVR benchmarks, including a 4.0 R@1 improvement on MSR‑VTT. The code is publicly available on GitHub.

By Uicheol Jung, Juyoung Hong, Hojung Kwon, Yukyung Choi

multimodal benchmarks safety

arXiv AI

Jun 2

CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects

arXiv:2510. 14904v4 Announce Type: replace-cross Abstract: Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language.

By Gabriel Fiastre, Antoine Yang, Cordelia Schmid

computer-vision multimodal benchmarks

Related stories