← Back to all news
arXiv Computer Vision

Image Recognition with Vision and Language Embeddings of VLMs

The paper evaluates how vision-language models (VLMs) perform in image classification tasks, comparing language-guided and vision-only approaches across various dual-encoder VLMs such as SigLIP 2 and RADIOv2.5 on ImageNet-1k and its label-corrected variant. It identifies key factors influencing accuracy—including prompt design, class diversity, k‑NN neighbors, and reference set size—and demonstrates that language and vision provide complementary strengths. A simple, learning‑free fusion method based on per‑class precision is proposed to further improve classification performance.

Related stories

arXiv Computer Vision
5d ago

Multimodal Large Language Models as Image Classifiers

The paper investigates how evaluation protocols and ground‑truth quality affect the classification performance of Multimodal Large Language Models (MLLMs). It identifies and corrects key issues such as discarded out‑of‑list outputs, inflated distractor choices, and poor open‑world mapping, and shows that design choices like batch size, image ordering, and text encoder selection significantly influence accuracy. Using a multilabel reannotation of 625 ImageNet‑1k classes (ReGT), the study finds that corrected labels can boost MLLM performance by up to 10.8%, narrowing the gap with supervised models, and demonstrates that MLLMs can assist human annotators in about half of difficult cases.

By Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas
llmsmultimodal
More like this →
arXiv AI
Aug 20

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

arXiv:2608. 18339v1 Announce Type: cross Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.

By Qi Yu, Zhichen Zeng, Katherine Tieu, Xiyuan Yang, Ruizhong Qiu, Yuchen Yan, Lihui Liu, Yanjun Zhao, Lingjie Chen, Jingrui He, Hanghang Tong
llmsragcomputer-visionmultimodalbenchmarkssafety
More like this →
arXiv AI
Jun 24

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

arXiv:2606. 24716v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence.

By Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Beg\"um Demir
llmsragbenchmarkssafety
More like this →
arXiv Machine Learning
Sep 30

Are In-Context Images Worth 10 Dimensions?

arXiv:2609.37659v1 Announce Type: cross Abstract: There has been significant work on understanding the In-Context Learning capabilities of Large Language Models, especially on the induction circuit....

By Adhemar de Senneville, Xavier Bou, J\'er\'emy Anger, Rafael Grompone, Gabriele Facciolo
llmscomputer-vision
More like this →
arXiv Computer Vision
Aug 28

G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

The paper introduces G2D, a training‑free framework that combines a discriminative model (CLIP) for broad candidate retrieval with a generative vision‑language model for fine‑grained, image‑grounded verification. By using CLIP’s top‑K shortlist and a structured prior from candidate names and probabilities, G2D focuses generative reasoning on uncertain samples, achieving an average accuracy of 68.85% across eight benchmarks—higher than both CLIP alone (59.35%) and the standalone generative model (63.11%). The approach also adapts to various generator configurations and extends to other models such as DCLIP, WaffleCLIP, and CuPL.

By Zehua Hao, Fang Liu, Qinliang Wang, Yaoyang Du, Xinyan Huang, Puhua Chen
llmscomputer-visionmultimodalbenchmarks
More like this →