arXiv AI November 6, 2026 By Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar, Yonghuai Liu

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Read the original on arXiv AI →

SD‑LoRA introduces a training‑time structural distillation framework that embeds token‑level graph reasoning into a lightweight LoRA‑adapted CLIP model for few‑shot vision‑language adaptation. During training, a heterogeneous graph teacher captures patch‑patch, global‑local, and visual‑text relations over high‑resolution ViT tokens, and its structural bias is transferred to the CLIP‑LoRA backbone via implicit distillation and Prototype Predictive Alignment. At inference, the graph teacher is discarded, requiring only a single CLIP‑LoRA forward pass and a cache‑based classifier, yielding significant accuracy gains across 11 few‑shot benchmarks without extra computation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Sep 10

On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data

arXiv:2609.10321v1 Announce Type: new Abstract: Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in curren...

By Hongyuan Zhang, Xianda Guo, Yanlun Peng, Qianlong Yang, Yubin Guo, Pinhan Fu, Mulin Chen, Xiaozhen Qiao, Ping Luo

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 21

DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

arXiv:2607. 17467v1 Announce Type: cross Abstract: Few-shot Test-Time Domain Adaptation (FSTT-DA) seeks to adapt models to novel domains using only a handful of unlabeled target samples.

By Siobhan Reid, Zhixiang Chi, Li Gu, Omid Reza Heidari, Ziqiang Wang, Yang Wang

fine-tuning benchmarks

arXiv AI

Oct 27

Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization

arXiv:2505.07675v3 Announce Type: replace-cross Abstract: Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Rece...

By Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Sung Ju Hwang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Oct 5

Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

arXiv:2605.10247v2 Announce Type: replace Abstract: Applying Large Language Models (LLMs) to graph-structured data usually involves multi-step pipelines in which textual node attributes are compresse...

By Dario Vajda

llms fine-tuning benchmarks

arXiv AI

Jun 30

Spanning the Visual Analogy Space with a Weight Basis of LoRAs

arXiv:2602. 15727v2 Announce Type: replace-cross Abstract: Visual analogy learning enables image editing via demonstration rather than textual description, allowing users to specify complex transformations difficult to articulate in words.

By Hila Manor, Rinon Gal, Haggai Maron, Tomer Michaeli, Gal Chechik

diffusion robotics fine-tuning benchmarks

arXiv Computer Vision

Aug 27

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

MLLMCLIP introduces a heterogeneous distillation framework that transfers multimodal knowledge from a generative Multimodal Large Language Model (MLLM) teacher directly into a discriminative CLIP student, eliminating the need for synthetic hard negatives. The method uses an attention-based per-layer token selection and a CKA-based distillation loss to bridge architectural differences between the two models. As a result, MLLMCLIP achieves state‑of‑the‑art compositional accuracy and improves zero‑shot classification and image‑text retrieval performance.

By Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji

llms diffusion efficiency multimodal benchmarks