← Back to all news
arXiv Computer Vision

Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval

The paper introduces Intrinsic Temporal Adaptation (ITA) for Partially Relevant Video Retrieval (PRVR), a task that seeks untrimmed videos containing moments relevant to a text query. ITA employs a Backbone-Internal Temporal Adaptation that lets the final visual transformer layers attend to neighboring frames, creating temporally aware embeddings while keeping CLIP frozen. Additionally, an Affinity-Weighted Gradient Propagation technique softly aggregates top‑k frames based on text‑frame affinities to better handle the weakly supervised nature of PRVR, leading to state‑of‑the‑art performance and more accurate frame‑level evidence retrieval.

Related stories

Hugging Face Trending Papers
Sep 2

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

TAME introduces a Temporal-Aware Mixture-of-Experts framework for Text-Video Retrieval that enhances CLIP-based models by incorporating frame-level structure and temporal relations. It adds sparse Mixture-of-Experts layers with frame-consistent routing, Frame-Temporal tokens for global cross-frame aggregation, and a Cross-Temporal Interaction and Aggregation module to refine sentence-video similarities. Experiments on multiple TVR benchmarks show consistent performance gains, such as a 4.0 R@1 improvement on MSR‑VTT over CLIP4Clip.

multimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Sep 3

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

TAME is a CLIP‑based framework for Text‑Video Retrieval that incorporates temporal modeling through three key innovations: sparse Mixture‑of‑Experts layers with frame‑consistent routing, Frame‑Temporal tokens that aggregate cross‑frame information, and a Cross‑Temporal Interaction and Aggregation module for refining frame‑wise similarities. These components enable the model to capture both local visual patterns and long‑range temporal dependencies, leading to consistent performance gains over CLIP‑based baselines on multiple TVR benchmarks, including a 4.0 R@1 improvement on MSR‑VTT. The code is publicly available on GitHub.

By Uicheol Jung, Juyoung Hong, Hojung Kwon, Yukyung Choi
multimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Aug 31

Training-Free Temporal Abstraction for General Video Understanding

The paper introduces STITCH, a training‑free method that partitions videos into semantically meaningful temporal chunks using a frozen video‑text backbone. By detecting changes in the embedding sequence of short video windows, STITCH produces reusable temporal abstractions that can be applied to multiple tasks such as event boundary detection, language‑based moment retrieval, and frame selection for vision‑language models. Experiments show that STITCH performs competitively with specialized methods while requiring no task‑specific training, especially when processing is limited to a few frames or tokens.

By Etienne Casanova, Sevan Brodjian, Pietro Perona
llmsragmultimodal
More like this →
arXiv Computer Vision
Sep 3

Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings

The paper introduces AllocEmbed, an allocate‑then‑embed framework that reallocates a fixed visual‑input budget across more video frames to improve retrieval performance. A lightweight allocator uses low‑cost previews to assign frame‑wise resolutions before the embedding backbone, preserving detail where it most benefits retrieval while reducing visual cost elsewhere. Retrieval‑Driven Policy Optimization (RDPO) learns the allocator directly from retrieval feedback, and the method integrates with existing systems without modifying the embedding model.

By Song Jin, Zhongtao Jiang, Chenglei Shen, Huanxuan Liao, Haozhe Chi, Zhiwei Wang, Kun Xu, Yong Liu
ragbenchmarks
More like this →
arXiv Computer Vision
Oct 21

SuperClip Pyramid for Video Temporal Grounding

arXiv:2610.26925v1 Announce Type: new Abstract: With the rapid expansion of online video content, video temporal grounding has become essential for localizing moments based on user-specific queries,...

By Sanghyeok Lee, Juyeon Ko, Joonmyung Choi, Jongha Kim, Dohwan Ko, Ji Soo Lee, Hyunwoo J. Kim
benchmarks
More like this →
arXiv AI
Oct 16

Where, How, and When: Query-Guided Grounding for Video Understanding

The paper introduces query‑guided grounding methods for video understanding, covering object‑level, relationship‑level, and moment‑level tasks. For object grounding, QDETRv and TOViT enable one‑shot and few‑shot localization of unseen objects using query‑context fusion, memory modules, and language‑aligned vision encoders. Relationship grounding is addressed with a few‑shot referring framework that optimizes a T‑partite random field, while moment grounding is tackled by the Moment Alignment Transformer (MATR), which employs dynamic time warping and self‑supervised pretraining for video‑to‑video and text‑to‑video retrieval.

By Yogesh Kumar
llmsragdiffusioncomputer-visionsafety
More like this →