← Back to all news
arXiv Computer Vision By Tao Yang, Jianying Zhou

Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders

Read the original on arXiv Computer Vision →

The paper investigates whether the similarity score (victim‑space target alignment, VTS) reported by adversarially attacked vision‑language models actually indicates that the target text can be recovered by an independent model. Using a new measurement instrument that includes supervised judges, blend controls, shuffled negatives, and a reference level, the authors conduct two preregistered studies across six contrastive encoders and three perturbation budgets. Robustly trained encoders (FARE, TeCoA, PMG, TRADES) transfer more independent evidence than vanilla CLIP or SigLIP, yet none reach the blend‑derived reference level, leaving practical recovery uncertain; VTS is only informative within a fixed robust encoder, prompting the authors to propose a new reporting protocol.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Oct 5

Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally

The paper demonstrates that a targeted adversarial perturbation can reduce a vision‑language model’s training loss to near zero for a fixed target caption, yet the same model, when generating freely, still produces the correct description. This phenomenon, termed the train/inference gap, is traced to a single autoregressive step where the target token’s rank is fixed across all images, and further analysis shows that the language decoder, rather than the visual encoder, determines whether the corrupted signal is amplified or suppressed. The study uses a controlled two‑stage PGD attack on Qwen2.5‑VL‑7B‑Instruct and evaluates the effect on 200 held‑out COCO images, revealing that adversarial robustness in autoregressive VLMs largely depends on the language decoder’s prior. whyItMatters":"The findings suggest that defenses and faithfulness evaluations for deployed vision‑language models should focus on the language decoder rather than the visual encoder, as the former is the key determinant of robustness to adversarial perturbations."

By Arun Josephraj Arokiaraj, Zekun Wu, Adriano Koshiyama
llmsmultimodalsafety
More like this →
arXiv Machine Learning
Oct 16

Purify Before You Align: Improving Representation Alignment for Diffusion Models

The paper proposes Local Appearance Purification (LAP), a method that refines the alignment target for diffusion models by removing locally predictable content and focusing on the residual. LAP improves generation quality across multiple pretrained encoders, notably reducing FID from 53.1 to 42.3 on ImageNet-256 with MAE. The authors provide a theoretical framework linking alignment effectiveness to the information emphasized by the target and introduce a constant‑readout test to predict purification benefits.

By Yingheng Wang, Yaoqiang Li, Yaqin Wu, Junwen Bai, Jiatao Gu, Christopher De Sa, Volodymyr Kuleshov
diffusionsafety
More like this →
arXiv AI
Sep 18

Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

The paper introduces a method for selecting the best vision‑language model for a downstream task by analyzing the internal dynamics of the visual encoder. It represents each task with layer‑wise conductance and uses an entropy‑regularized alignment to derive a target‑conditioned block importance distribution. The proposed Directional Conductance Divergence (DCD) metric captures asymmetric transferability, enabling accurate prediction of model rankings without direct inference, and achieves a 14.7% NDCG@5 improvement over SWAB on 48 VLMs across 21 datasets.

By Wei Yang, Hong Xie, Tao Tan, Xin Li, Defu Lian, Enhong Chen
llmsmultimodalbenchmarkssafety
More like this →