arXiv Computer Vision October 23, 2026 By Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, Dhruv Bansal

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

Read the original on arXiv Computer Vision →

The paper investigates how to best integrate large language model (LLM) guidance into visual question answering (VQA) systems. By prompting the 70‑billion‑parameter Llama 3.3 model to identify visual concepts, distractors, and external facts for each question, the authors inject this guidance into a frozen Vision Transformer (ViT) and a T5 text‑to‑text model via two methods: decision‑level late fusion and attention‑matrix injection. On the OK‑VQA benchmark, the shallow late‑fusion approach outperforms the deeper attention injection, achieving 39.4% exact match versus 31.8%, and a paraphrasing pipeline further expands the training set tenfold, indicating that the depth of integration should match the data regime.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI

Aug 7

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

arXiv:2604. 01280v2 Announce Type: replace-cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence.

By Marco Morini, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

llms nlp multimodal benchmarks

arXiv AI

Jun 16

When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs

arXiv:2602. 00344v2 Announce Type: replace-cross Abstract: While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens.

By Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li

llms rag multimodal

arXiv Computer Vision

Oct 2

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

arXiv:2610.01180v1 Announce Type: new Abstract: Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently str...

By Yuliang Cai, Mohammad Rostami, Jesse Thomason

llms nlp multimodal benchmarks

arXiv Machine Learning

Jul 9

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

arXiv:2607. 07179v1 Announce Type: cross Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents.

By Miguel Lopez-Duran, Elena Marrero, Julian Fierrez, Marta Robledo-Moreno, Ruben Vera-Rodriguez, Daniel DeAlcala, Aythami Morales, Ruben Tolosana, Oscar Delgado, Alvaro Ortigosa, Javier Ortega-Garcia

llms nlp fine-tuning multimodal

Hugging Face Trending Papers

Jul 8

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored.

llms nlp fine-tuning multimodal

arXiv Computer Vision

Sep 15

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End

arXiv:2609.13815v1 Announce Type: new Abstract: Text-centric Visual Question Answering (VQA) requires reading and reasoning over text embedded in images, a task made substantially harder when images...

By Ritali Vatsi, Rachapudi Jagadeesh, Shruti Singh Baghel, Himani Sharma, Amit Shukla, Pawan Goyal

llms computer-vision nlp fine-tuning multimodal

One email a morning, machine-written

Related stories

Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering

When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

Beyond OCR Accuracy: Text-Centric VQA Under Image Degradation with Modular and End-to-End