arXiv AI

Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment

The paper introduces RoSA, a saliency-aware fine‑tuning framework for long‑text image‑text alignment that addresses the failure mode of saliency collapse. RoSA decomposes image‑text pairs into salient and contextual views using a modality‑independent strategy that combines visual saliency estimation with LLM‑based text decomposition, and employs auxiliary objectives to align these views. The method improves long‑text retrieval performance and robustness to saliency collapse, and its auxiliary objectives can be added to existing tuning methods for consistent gains.

arXiv Computer Vision

Sep 28

Amplify What You Gaze At: Target Saliency Boosting in Text-to-Image Generation

The paper introduces Target Saliency Boosting, a new task that enhances the visual prominence of a specific object in text-to-image generation without visual priors. It proposes GazeME, a lightweight framework that inserts learnable marker tokens around object descriptions to indicate which objects to emphasize or suppress. By building a saliency-semantics dataset and using Saliency Prior Marker Activation, GazeME learns to adjust markers during training and automatically applies them at inference, effectively boosting target saliency while maintaining semantic alignment and image quality.

By Shengqi Dang, Zhengxi Yu, Feilin Han, Xingyu Lan, Nan Cao

diffusion safety

arXiv AI

Jul 3

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

arXiv:2511. 01390v2 Announce Type: replace-cross Abstract: Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications.

By Xinyu Mao, Junsi Li, Haoji Zhang, Yu Liang, Ming Sun

llms diffusion nlp multimodal safety

arXiv Computer Vision

Oct 7

Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling

arXiv:2610.07689v1 Announce Type: new Abstract: Vision-Language Models (VLMs) such as CLIP excel in global semantic alignment but often lack fine-grained perceptual capabilities. This hinders dense p...

By Juntong Li, Lingwei Dang, Haomin Wu, Ziyan Qiu, Qingxin Xiao, Qingyao Wu

llms rag efficiency multimodal safety

arXiv AI

Sep 30

TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning

arXiv:2609.37581v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visu...

By Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li

llms efficiency multimodal benchmarks

arXiv Computer Vision

Sep 28

OpenVAM: Open-World Visual Attention Modeling with VLMs

OpenVAM is a new framework for visual attention modeling that combines a dense saliency map with language‑based explanations. It uses a decoupled design: a visual pathway for precise localization and a vision‑language head that generates grounded what/why explanations. The method is trained in three stages to preserve localization while adding language grounding, and a scalable pipeline creates multi‑domain annotations for evaluation.

By Kiana Hooshanfar, Amirhossein Kazerouni, Alireza Hosseini, Michael Brudno, Babak Taati

robotics safety

arXiv Computer Vision

Sep 15

Rank-Consistent Set Reasoning for Co-Salient Object Detection

arXiv:2609.13706v1 Announce Type: new Abstract: Co-salient object detection (Co-SOD) requires a model to find foreground regions that are salient in individual images and supported by the image group...

By Yuan Xiang, Matteo Rossi, Yingzhou Chen

computer-vision

arXiv Machine Learning

Jun 2

Saliency-Aware Model Merging

arXiv:2606. 00511v1 Announce Type: new Abstract: Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency.

By Jungin Park, Jiyoung Lee, Kwanghoon Sohn

fine-tuning efficiency

arXiv AI

Jul 28

Disentangling Semantic Attention from Structural Bias in the Attention Manifold

arXiv:2607. 24017v1 Announce Type: cross Abstract: The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws.

By Pengkun Jiao, Bin Zhu, Jingjing Chen, Yu-gang Jiang

llms multimodal benchmarks safety

arXiv Machine Learning

Oct 7

DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

arXiv:2610.08341v1 Announce Type: cross Abstract: Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redu...

By Shuo Yang, Changbai Li, Linlin Yang, Huobin Tan, Rongyu Chen, Tongfei Chen, Tian Wang, Sheng Xu, Baochang Zhang

llms efficiency multimodal benchmarks

arXiv Computer Vision

Aug 24

Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection

Crane is a CLIP‑based framework for zero‑shot anomaly detection that enhances dense localization by adapting the vision encoder with a correlation‑based attention module and conditioning learnable prompts on global image context. It further fuses anomaly‑relevant patch features into the global representation for more sensitive image‑level detection, and a variant called Crane+ leverages DINOv2 spatial correlations for stronger pixel‑level performance. Across seven industrial benchmarks, Crane raises mean image‑level AP by 4.5% and Crane+ boosts mean pixel‑level AUPRO by 9.0%.

By Alireza Salehi, Mohammadreza Salehi, Reshad Hosseini, Cees G. M. Snoek, Makoto Yamada, Mohammad Sabokrou

multimodal benchmarks safety

Hugging Face Trending Papers

Jul 15

Fine-grained CLIP fine-tuning with self-annotated region alignment

Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme.

llms fine-tuning multimodal safety

arXiv AI

Jun 8

Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models

arXiv:2605. 20950v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference.

By Yulin Zhao, Zheng Zhang

llms multimodal benchmarks

Related stories