This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations. Existing methods often suffer from noisy pseudo-masks, limited visual-textual grounding, and difficulty handling synonyms or out-of-vocabulary (OOV) words.
PANORAMA introduces a new panoptic grounded captioning framework that jointly generates detailed image captions and associates each phrase with precise pixel-level masks. The authors create PanoCaps, a human‑annotated benchmark with dense captions and near‑complete pixel coverage, and propose a phrase‑mask matching protocol with a generalized Panoptic Quality metric. PANORAMA conditions a pretrained segmenter on contextualized phrase representations, learns to select appropriate masks, and achieves state‑of‑the‑art grounding performance on PanoCaps and other pixel‑level tasks.
By Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid
Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image ca...
arXiv:2608. 07886v1 Announce Type: cross Abstract: Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region.
By Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna
arXiv:2607. 11008v1 Announce Type: cross Abstract: Open-vocabulary dense perception (OVDP) aims to localize objects unseen during training by leveraging textual knowledge.
By Mingjie Xie, Guangjun He, Dongli Xu, Youtian Lin, Hongjue Li, Pengming Feng, Jian Guan, Yue Deng
arXiv:2608.30621v1 Announce Type: cross
Abstract: Collecting natural-language referring expressions along with region annotations, such as masks or boxes, is a major bottleneck in visual grounding (V...
By Junbeom Hong, Seonghoon Yu, Hyung Rok Jung, Sundong Kim, Jeany Son