VidOmni-Bench is a new benchmark for fine‑grained video understanding that asks models to verify whether each event in dense video captions is supported by the video. It contains 500 videos covering five complexity types and durations from 4 seconds to 90 minutes, and uses human‑verified sentence‑level labels to create hard negatives. Experiments show that Video‑LLMs often hallucinate events, struggle to detect incorrect descriptions, and exhibit varying weaknesses depending on video complexity and duration.
By Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim
arXiv:2609.09973v1 Announce Type: new
Abstract: Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generate...
By Zizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li, Yang Lu, Meng Cao, Ping Huang, Xiaoming Simon Wang
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set.
arXiv:2606.29997v2 Announce Type: replace
Abstract: Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limi...
By Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Daichi Yashima, Komei Sugiura
The paper presents a batik-captioning system that uses class labels as compact supervision when expert text annotations are scarce. It combines a LoRA‑adapted DINOv2 classifier with a Qwen2.5‑VL‑7B captioner, both loaded in 4‑bit NF4, and evaluates three captioner conditions across 150 images, achieving a 0.9067 ground‑truth class‑label mention rate. The classifier attains 96.10% pooled accuracy, with higher performance on public‑source images (98.33%) than on local images (75.76%), and the study highlights how class conditioning preserves label information while also propagating classification errors into fluent captions.
By Novanto Yudistira, Muhammad Dzaka Aufa Fadhillah, Irawati Nurmala Sari
arXiv:2609.36628v1 Announce Type: new
Abstract: Vision-Language Models (VLMs) can generate rich video captions, yet often misidentify which person performs an action or which limb is involved, partic...
By Yanan Wang, Tingsong Li, Kaixun Jiang, Chongyang Zhong, Chenwei Xoe, Zhaohe Liao
Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected objects to curb hallucination, but apply only a single, fixed correction pass without prioritizing which objects matter most, leaving semantically significant content omitted.
The paper introduces a position‑aware rubric that breaks down long image captions into atomic visual claims, allowing verification of generated text against these claims and revealing distinct failure modes: unsupported specificity in the middle and vague language at the end. Using this diagnosis, the authors propose PAGER, a reinforcement learning objective that provides separate rewards for middle evidence, tail continuation, claim coverage, and length control. Experiments on detailed‑caption benchmarks show that PAGER improves position‑wise grounding and pairwise caption preference while maintaining overall caption quality.
By Yuanze Hu, Zhichao Yang, Junwei Jing, Xin Yang, Wenxuan Liu, Tinghai Zhang, Jia Fu, Hongzhi Zhang, Xiangyu Wu, Ruiming Tang, Tingting Gao, Han Li
The paper examines the intermediate‑entropy regime of natural language generation, where a single input can have multiple valid outputs such as in peer‑review generation and image captioning. It introduces a target‑side consolidation method that merges all references into a single target preserving every distinct information unit, and presents the MERC‑67K dataset of 36,537 papers and 31,014 images with both original and consolidated references. Human evaluations show that models trained on consolidated references achieve higher coverage and overall quality scores, outperforming multi‑reference baselines across several large language models while using fewer individual references during training.
By Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary, Manish Shrivastava, Vasudeva Varma
DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.
By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
arXiv:2610.33798v1 Announce Type: cross
Abstract: An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still ju...
By Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam Penamakuri
arXiv:2607. 15216v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) often introduce errors when generating image captions, resulting in misaligned image-text pairs.
By Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz