arXiv Machine Learning
Jul 2
AdaBoosting Text Prompts for Vision-Language Models
arXiv:2607. 00684v1 Announce Type: new Abstract: The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts.
ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction
ReViCo (Real Visual Correction) is a new benchmark that tests Vision Language Models (VLMs) on the task of correcting text errors in real‑world images, requiring deep understanding of visual text and its context. The study evaluates VLMs using both prompt‑based and targeted training approaches, revealing a significant performance gap between current models and humans. The results show that most VLMs struggle to accurately perceive visual text, leading to frequent correction mistakes, thereby underscoring the need for more robust, text‑aware VLMs.
Related stories
arXiv AI
Jun 24
Listening makes Vision Clear for VLMs
arXiv:2606. 23763v1 Announce Type: cross Abstract: Recent work typically assesses vision--language consistency using attention distributions of answer-side tokens.
arXiv AI
Jun 19
The Hidden Evolution of Disguised Visual Context inside the VLM
arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.
arXiv Machine Learning
Jul 9
Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
arXiv:2607. 07179v1 Announce Type: cross Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents.
Hugging Face Trending Papers
Jul 8
Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored.
arXiv AI
Jul 28
Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
arXiv:2607. 24354v1 Announce Type: new Abstract: Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results.
arXiv AI
Aug 18
MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4
arXiv:2406. 00971v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have recently seen significant advancements through integrating with Large Language Models (LLMs).
arXiv AI
Aug 25
HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
arXiv:2509.23928v3 Announce Type: replace-cross Abstract: Speculative decoding has proven effective for accelerating inference in Large Language Models (LLMs), yet its extension to Vision-Language Mo...
Hugging Face Trending Papers
Jul 22
Test-Time Training for Modality Order Consistency in Vision-Language Models
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure.
Hugging Face Trending Papers
Aug 12
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see.
Hugging Face Trending Papers
Aug 27
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
The paper demonstrates that vision‑language models (VLMs) possess a small set of attention heads, called Visual Retrieval Heads (VRHs), that are causally responsible for linking text prompts to specific image regions. By adapting head‑scoring techniques from language models, the authors identify VRHs as the heads whose attention from output prediction tokens, summed over the ground‑truth referent region, most reliably indicates causal grounding. Experiments across eleven VLMs and five referring‑expression benchmarks show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect, and that VRHs generalize across diverse visual tasks and transfer across models sharing an LLM backbone.
arXiv AI
Aug 13
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
arXiv:2608. 11847v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning.