The paper introduces Cue2Narrate, a two‑stage pipeline that jointly predicts what visual events to narrate and when to insert the narration in long, untrimmed movie clips. It uses a dual‑head audio‑visual localizer to identify visual cue and narration windows, followed by a LoRA‑adapted vision‑language model that generates concise audio descriptions, trained with a Description Ranking Loss. The authors also present the LongLSMDC benchmark, comprising up to 8‑minute clips, and show that Cue2Narrate outperforms video‑only and audio‑only baselines by 5–12 points in average mAP and improves AD generation over fine‑tuned base VLMs.
By Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach
Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visua...
TiTok is an audio‑visual large language model designed for multi‑segment temporal grounding in untrimmed videos. It introduces Time Token Interleaving (TTI) to align temporal perception with prediction, and uses decoupled reinforcement‑learning rewards (global, local, count, precision, format) optimized via GDPO. Evaluated on a new UnAV‑100 protocol with the CountF1 metric, TiTok achieves state‑of‑the‑art results (65.7 mIoU, 0.58 CountF1).
By Eunji Shin, Dahyun Choi, Seungyeon Jo, Yejin Hong, Jiyoung Lee
arXiv:2609.10355v1 Announce Type: cross
Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...
By Killian Steunou, Yannis Tevissen, Moun\^im A. El Yacoubi
Video-HolmesV2 is a new benchmark that tests multimodal large language models on their ability to reason with spatio‑temporal audio‑visual evidence in long videos. It requires models to justify answers with precise evidence, uses a multi‑model cross‑verification pipeline and a spatio‑temporal evidence‑aware metric, and introduces an audio‑text guided token compression framework to reduce long‑context noise. In evaluations, even strong proprietary models score below 60% while the proposed approach outperforms comparable open‑source omni‑models.
By Zhaoyang Wei, Zipeng Wang, Yushe Cao, Chenhui Qiang, Shuaibing Cheng, Xuesong Yang, Sen Nie, Bowen Jiang, Wenchao Ding, Yanchao Hao, Zheng Wei, Xuehui Yu, Zhenjun Han
Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a te...
arXiv:2610.08192v1 Announce Type: new
Abstract: Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at...
By Souptik Sen, Zahra Ahmadi
arXiv:2608. 14016v1 Announce Type: cross Abstract: Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else.
By Mathew Varghese
Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient c...
LEAP is a framework for long audio‑video question answering that avoids encoding entire recordings by dividing them into fixed‑duration blocks. It performs a lightweight localization pass on each block to score short candidate windows, then pools the highest‑ranked windows for a single bounded answer pass, keeping the answer input and peak context independent of recording length. The method trains both a localization LoRA and an answer LoRA, supports causal streaming queries, and achieves significant performance gains over baseline models on multiple AVQA benchmarks.
By Juyi Lin, Zhiqiang Lao, Jiali Cui, Lin Zhao, Pu Zhao, Dichang Zhang, Arman Akbari, Yu Qi, Xinru Jiang, Yanzhi Wang, Heather Yu, Liang Peng
arXiv:2610.19419v1 Announce Type: new
Abstract: MLLM-based detectors formulate vision-language grounding as autoregressive coordinate generation, yet in dense scenes, they often collapse into a box r...
By Chenxu Wang, Hangyi Kuang, Bo-Wen Yin, Yuxuan Li, Quan-Sheng Zeng, Yang Zhang, Xu Guo, Xiang Li, Qibin Hou
arXiv:2603.16086v2 Announce Type: replace-cross
Abstract: While recent Vision-Language-Action (VLA) models have begun to incorporate audio, they typically treat sound as static pre-execution prompts...
By Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, Hesheng Wang