arXiv Machine Learning

KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale

arXiv:2607. 18885v1 Announce Type: new Abstract: Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility, using a fixed trade-off weight tuned on curated ImageNet-1K.

arXiv Machine Learning

Jul 28

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

arXiv:2501. 19060v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes.

By Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo

llms fine-tuning multimodal safety

arXiv AI

Jul 21

A${}^2$BM: Alignment-Aware Bridge Matching for Image-to-Image Translation

arXiv:2607. 16294v1 Announce Type: cross Abstract: Paired image-to-image translation underpins a wide range of computer vision tasks, including image editing, sensor translation, and domain adaptation.

By Aimi Okabayashi (UBS Vannes), Georges Le Bellier (LIP, CEDRIC - VERTIGO), Nicolas Audebert (LaSTIG, IGN, CEDRIC - VERTIGO), Charlotte Pelletier (OBELIX), Thomas Corpetti (LETG - Rennes), Nicolas Courty (OBELIX)

llms diffusion computer-vision fine-tuning safety

arXiv AI

Jul 28

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

arXiv:2607. 23271v1 Announce Type: cross Abstract: Contrastive vision-language models such as CLIP map semantically opposite phrases (e.

By Chen-Yi Lu, Yueh-Shao Chen, Somali Chaterji

llms rag diffusion fine-tuning multimodal safety

arXiv AI

Jun 29

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.

By Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv Machine Learning

Jul 23

SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation

arXiv:2605. 30116v2 Announce Type: replace-cross Abstract: Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models.

By Zhuguanyu Wu, Ruihao Gong, Yang Yong, Yushi Huang, Xiangyu Fan, Lei Yang, Dahua Lin, Xianglong Liu

diffusion efficiency safety

arXiv AI

Jun 24

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

arXiv:2606. 24716v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence.

By Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Beg\"um Demir

llms rag benchmarks safety

arXiv Machine Learning

Jun 26

The Geometry of Updates: Fisher Alignment at Vocabulary Scale

arXiv:2606. 27242v1 Announce Type: new Abstract: Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets.

By John Sweeney

llms rag nlp safety

arXiv Machine Learning

Jul 2

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

arXiv:2602. 21397v2 Announce Type: replace-cross Abstract: Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights.

By Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani

llms rag multimodal benchmarks safety

arXiv AI

Jun 2

Improving Visual Representation Alignment Generation with GRPO

arXiv:2606. 00583v1 Announce Type: cross Abstract: Recent diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak alignment between generative and discriminative representations.

By Shentong Mo, Sukmin Yun

llms rag diffusion safety

arXiv Machine Learning

Jul 10

Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

arXiv:2603. 12478v2 Announce Type: replace-cross Abstract: Multimodal instruction tuning is often compute-inefficient because training budgets are spread across large mixed image-video pools whose utility is highly uneven.

By Rujie Wu, Haozhe Zhao, Hai Ci, Yizhou Wang

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 22

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

arXiv:2607. 18615v1 Announce Type: cross Abstract: Machine unlearning for vision-language models (VLMs) remains underexplored.

By Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

llms multimodal

Related stories