The paper investigates whether language prompts selected by zero‑shot accuracy remain effective after visual adaptation in source‑free cross‑domain few‑shot learning. Using a paired protocol, the authors compare generic class‑name templates with detailed class descriptions before and after Low‑Rank Adaptation (LoRA) on datasets such as EuroSAT, CropDisease, ISIC, and ChestX. They identify two regimes: semantic saturation, where detailed prompts are already useful before adaptation, and semantic emergence, where detailed prompts become more useful only after visual representation updates, driven by changes in prediction patterns.
arXiv:2607. 18695v1 Announce Type: cross Abstract: A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors.
The paper introduces an experiment where a small masked language model (DeBERTa) is initialized with visual embeddings for tokens that correspond to image regions, following St. Augustine’s ostensive definition of word learning. The visual initialization leaves a measurable imprint that persists through training, yet it does not improve performance on most BabyLM benchmarks that test abstract grammatical knowledge. However, the seeded models show a consistent advantage in zero‑shot object‑property tasks and in a custom Visual‑Property Swap benchmark that probes color, material, size, and shape knowledge, with the advantage confined to the seeded words and transferable to newly seeded words.
arXiv:2610.34610v1 Announce Type: new Abstract: Vision-language anomaly detectors describe normal and abnormal appearance with text, and their gains are commonly credited to the semantic content of t...
arXiv:2608. 02830v1 Announce Type: cross Abstract: Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied.
arXiv:2607. 28967v1 Announce Type: cross Abstract: Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch.
The paper introduces a perception interface that separates vision from language in vision‑language models. A frozen perception stack detects objects, a deterministic semantic serializer converts the perceived state into text, and a standard text‑only large language model (LLM) answers questions. Experiments on a campus‑robot benchmark show that this serialized interface outperforms a zero‑shot VLM of the same language‑model size, especially as the language model shrinks, and that the advantage persists under paraphrase and different supervision regimes.
DVLA-RL++ introduces Dual-Level Vision‑Language Alignment with Reinforcement Learning Gating for few‑shot learning. It adds complementary semantic purification (CSP) to separate intrinsic from nuisance textual descriptions and counterfactual reinforcement‑learning gating (CRG) to learn layer‑wise fusion strengths. Experiments on standard, fine‑grained, and cross‑domain benchmarks demonstrate state‑of‑the‑art accuracy, improving on DVLA‑RL by an average of 1.4%.
The paper investigates how much semantic information is lost when frozen foundation models are combined for few‑shot 3D segmentation. By varying the number of retained semantic alternatives before fusion, the authors show that keeping the full distribution of class scores yields higher harmonic‑mean IoU than collapsing to a single class. Experiments on ScanNet200 and ScanNet++ confirm that full‑distribution fusion consistently outperforms top‑1 and other operators, and that most useful information is recovered by retaining a compact set of plausible alternatives.
The paper explores soft prompting for few‑shot object detection with vision‑language models, showing that optimizing a small number of continuous prompt tokens—especially when placed at the cross‑modal boundary and initialized from an empty space token—can match LoRA performance while training far fewer parameters. Soft prompting also avoids catastrophic forgetting, transfers to newer models, and can be verbalized into readable prompts. The study extends these findings to manipulation tasks, indicating that VLMs already contain much of the necessary knowledge for specialized domains, and the main challenge is learning how to ask for it.
arXiv:2609.10321v1 Announce Type: new Abstract: Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in curren...
The paper revisits the use of contrastive loss for post‑training CLIP models, showing that the reported catastrophic forgetting is mainly due to an overly large temperature parameter in the InfoNCE objective. By setting a smaller temperature, contrastive post‑training can actually improve CLIP. The authors introduce ComCLIP, a lightweight single‑epoch recipe that freezes the text encoder, trains the vision encoder with a properly tempered contrastive loss, an MSE anchoring loss, and a relational distillation loss, achieving comparable zero‑shot classification to CLIP‑Refine while significantly enhancing visual feature transferability and maintaining downstream compatibility with LLaVA.
arXiv:2608.23903v1 Announce Type: new Abstract: Visual foundation models are commonly adapted under the assumption that the appearance of incoming data may change while the semantic meaning of the pr...