Hugging Face Trending Papers

Repurposing CLIP to Localize at Pixel Level

Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases.

Hugging Face Trending Papers

Jul 15

Fine-grained CLIP fine-tuning with self-annotated region alignment

Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme.

llms fine-tuning multimodal safety

arXiv AI

Jul 31

TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

arXiv:2607. 26107v1 Announce Type: cross Abstract: Dense vision-language understanding, including object localization, region recognition, and open-vocabulary semantic segmentation, requires associating language concepts with spatially grounded visual regions.

By Xinran Liu, Shouqian Shi, Yutong Chen, Ge Wang, Xin-Wei Yao, Sheng Zhong

rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 2

A Lightweight Context-Driven Training-Free Network for Scene Text Segmentation and Recognition

arXiv:2503. 15639v2 Announce Type: replace-cross Abstract: Modern scene text recognition systems often depend on large end-to-end architectures that require extensive training and are prohibitively expensive for real-time scenarios.

By Ritabrata Chakraborty, Shivakumara Palaiahnakote, Umapada Pal, Cheng-Lin Liu

computer-vision benchmarks

arXiv Machine Learning

Jul 20

Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

arXiv:2502. 06818v4 Announce Type: replace Abstract: Recent works modify CLIP to perform open-vocabulary semantic segmentation in a training-free manner (TF-OVSS).

By Jingyun Wang, Cilin Yan, Guoliang Kang

rag computer-vision benchmarks

arXiv AI

Jun 2

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

arXiv:2606. 00987v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored.

By Bingyu Li, Da Zhang, Tao Huo, Zhiyuan Zhao, Junyu Gao, Xuelong Li

llms agents computer-vision fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Jul 30

One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting

Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances.

llms rag computer-vision reinforcement-learning multimodal safety

arXiv Machine Learning

Jun 2

Domain Adaptation with a Single Vision-Language Embedding

arXiv:2410. 21361v2 Announce Type: replace-cross Abstract: Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in real-world autonomous driving scenarios, especially under rare or adverse conditions.

By Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick P\'erez, Raoul de Charette

rag agents computer-vision robotics fine-tuning multimodal

arXiv AI

Jun 2

CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models

arXiv:2605. 13178v2 Announce Type: replace-cross Abstract: In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead.

By Sangin Lee, Yukyung Choi

llms fine-tuning efficiency multimodal

arXiv AI

Jul 1

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

arXiv:2606. 31603v1 Announce Type: cross Abstract: Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.

By Nikolai R\"ohrich, Julian Glei{\ss}ner, Ahmed H. A. Ibrahim, Silvan Mertes, Tobias Huber

agents diffusion computer-vision fine-tuning efficiency safety

arXiv AI

Aug 11

MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

arXiv:2608. 09818v1 Announce Type: cross Abstract: Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding.

By Haoyu Yang, Meixing Shi, Zengjie Chen, Haoran Sun, Haitao Leng, Xiaoming Shi, Yuxiang Cai, Yankai Jiang

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 3

ProCal: Inference-Time Proposal Calibration for Open-Vocabulary Object Detection

arXiv:2607. 01759v1 Announce Type: cross Abstract: Open-vocabulary object detection aims to localize and classify objects beyond the fixed set of categories seen dur ing training.

By Jae-Ryung Hong, Ho-Joong Kim, Seong-Whan Lee

rag computer-vision multimodal

arXiv AI

Jun 2

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

arXiv:2603. 00171v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details.

By Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, Jie Zhang

llms multimodal benchmarks

Related stories