VidOmni-Bench is a new benchmark for fine‑grained video understanding that asks models to verify whether each event in dense video captions is supported by the video. It contains 500 videos covering five complexity types and durations from 4 seconds to 90 minutes, and uses human‑verified sentence‑level labels to create hard negatives. Experiments show that Video‑LLMs often hallucinate events, struggle to detect incorrect descriptions, and exhibit varying weaknesses depending on video complexity and duration.
By Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim
The paper introduces OACap, a caption-guided active observation system for detailed video captioning that requests targeted observations to fill missing information and iteratively refines its captions. OACap consistently improves performance on general video-understanding benchmarks while using only a text‑only reader. The authors also present FSCap‑Bench, a new evaluation suite with Core‑Support Factuality Score and Cross‑Modal Semantic Round‑Trip Score to assess factual and semantic preservation in captions.
By Hao Shi, Hong Yan, Xiaoyang Zhang, Qing Cui, Chao Huang
arXiv:2606.29997v2 Announce Type: replace
Abstract: Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limi...
By Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Daichi Yashima, Komei Sugiura
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set.
The paper introduces CodeCap-Bench, a benchmark for evaluating whether executable code can serve as an effective video caption. It compares code captions to natural language captions on tasks such as question answering and video reconstruction, finding that code captions yield higher reasoning accuracy and reconstruction fidelity. The study also assesses current multimodal large language models, noting that while proprietary models show promise, they still struggle with complex temporal dynamics, and open‑source models lag significantly.
By Jiafu Tang, Yifan Xu, Eric Tang, Junjie Zhao, Shipeng Luo, Te Sun, Haoxuan Hu, Yiyan Ji, Xinlong Chen, Zihao Wang, Yang Shi, Yuanxing Zhang, Pengfei Wan, Jiaheng Liu
DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.
By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
The paper introduces Redemption Score (RS), a multi‑modal evaluation framework for image captioning that combines three complementary signals: Mutual Information Divergence for global image‑text alignment, DINO‑based perceptual similarity of cycle‑generated images for visual grounding, and LLM text embeddings for contextual similarity to human references. RS fuses these signals to provide a more holistic assessment, achieving a Kendall‑τ of 58.42 on Flickr8k and outperforming most prior methods. The framework demonstrates consistent performance across Conceptual Captions and MS COCO, offering a robust evaluation that captures both visual accuracy and text quality.
By Ashim Dahal, Ankit Ghimire, Saydul Akbar Murad, Nick Rahimi
The paper introduces VGCap, a framework for assessing the quality of AI-generated videos that uses visually grounded captions instead of relying solely on source prompts. VGCap generates a global caption by minimally revising the source prompt with multi‑frame visual evidence and also produces prompt‑independent segment captions to describe localized temporal content. These captions jointly support evaluation of spatial quality, temporal quality, and text‑video alignment, and experiments on LGVQ and FETV datasets show VGCap outperforms existing baselines across all three dimensions.
By Kehao Zhou, Yixuan Li, Yuxuan Jiang, Fan Zhang
The paper introduces three new vision‑centric evaluation benchmarks—temporal frame retrieval, video future prediction, and causal memory distortion—to assess visual question answering in large video models. Unlike traditional benchmarks that rely on text-based multiple choice questions, these tasks require models to reason directly from visual inputs. The authors find that current state‑of‑the‑art models struggle with visual queries, highlighting a gap in visual understanding that future research should address.
By Rwiddhi Chakraborty (Oliver), Yinong (Oliver), Wang, Cheng Zhang, Fan Bai, Zhuoran You, Michael Kampffmeyer, Yong Jae Lee, Fernando De la Torre, Robert Jenssen
arXiv:2607. 08489v1 Announce Type: cross Abstract: Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention.
By Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, Ufuk Topcu
arXiv:2610.10163v1 Announce Type: new
Abstract: Linking people's appearance and actions to character identities is essential for understanding video narratives. We present a framework for identity-aw...
By Anas Filali Razzouki, Killian Steunou, Khalil Guetari, Thomas Kling, Moun\^im El-Yacoubi, Yannis Tevissen
Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected objects to curb hallucination, but apply only a single, fixed correction pass without prioritizing which objects matter most, leaving semantically significant content omitted.