arXiv AI
Aug 21
Question-Guided Evidence Acquisition for Multimodal Visual Question Answering
arXiv:2608. 19739v1 Announce Type: cross Abstract: Multimodal LLMs can see a document, but they often can't read it reliably.
Question-Guided Evidence Acquisition for Multimodal Visual Question Answering
Related stories
arXiv AI
Jul 20
Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models
arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?
arXiv AI
Oct 12
The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs
The paper investigates how visual interventions—training‑free edits applied to multimodal large language models—affect hallucinations and prediction quality. By applying a visual‑residual edit across successive decoder layers and matching its RMS strength, the authors identify an Evidence Anchoring Boundary (EAB) where the intervention’s influence drops sharply; effects before the EAB are 13.78–26.10× larger than after it. They further compare counterfactual subtraction across different question types and introduce calibrated selective routing, which only alters predictions when calibration supports an intervention, improving attribute accuracy by 1.56–1.81 percentage points while avoiding the harms of universal subtraction.
arXiv Computer Vision
Oct 26
The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets
arXiv:2610.32664v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) read images through visual tokens, and under the small per-image budgets that efficient deployment requires, t...
arXiv AI
Jun 16
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
arXiv:2606. 16494v1 Announce Type: cross Abstract: Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base.
arXiv Computation and Language
3d ago
VLMs Read Aloud: The Gap Between Recognizing and Using Visual Text
The paper investigates the disparity between a vision‑language model’s ability to recognize text in compressed images and its ability to use that text for answering questions, a phenomenon termed the read‑use gap. Experiments show that while models can transcribe images accurately, they often fail to answer based on the transcription, and that recognition saturates at lower pixel resolutions than usage. The authors propose ALOUD, a method that allocates visual resolution to evidence detection and language tokens to reasoning, which doubles the accuracy of a frozen Qwen3‑VL‑8B on heavily compressed documents without retaining the original text.
arXiv Machine Learning
Oct 5
Min-Cost Flow Routing for Evidence Assembly in Long Multimodal Documents
The paper introduces <flowreader>, a method that casts evidence selection for long multimodal documents as a minimum‑cost flow problem over a multimodal content graph. It uses spectral decomposition to identify latent query‑relevant aspects and allocates a fixed evidence budget proportionally to their spectral energy, ensuring aspect coverage without a language‑model planning call. On the VisDoMBench benchmark with Qwen3‑VL‑32B, <flowreader> achieves the highest macro accuracy (68.9%) and outperforms prior systems by 2.7 points, while using fewer content nodes and reader tokens.
arXiv AI
Sep 23
Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
arXiv:2609.22628v1 Announce Type: new Abstract: Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We iso...
arXiv AI
Aug 11
Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution
arXiv:2608. 07943v1 Announce Type: new Abstract: Multi-page visually-rich document understanding (MP-VRDU) requires managing evidence that is sparse, spread across pages, and often exceeds a model's context window.
arXiv AI
Jul 8
Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG
arXiv:2607. 05438v1 Announce Type: cross Abstract: Multimodal retrieval-augmented generation (RAG) grounds a generator in evidence drawn from heterogeneous modalities -- text, tables, and images.
arXiv Computer Vision
Aug 26
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
arXiv:2606.18974v3 Announce Type: replace Abstract: Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks. This incurs roughly a...
arXiv Computer Vision
Sep 21
From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists
The paper investigates how general vision‑language models (VLMs) develop specialized optical character recognition (OCR) capabilities. By applying a causal intervention protocol, the authors identify sparse, stable OCR‑head sets in several VLMs and show that these heads largely overlap with textual retrieval/copy heads found in general VLMs. The study concludes that full‑sequence OCR functions as a dense multimodal copy‑and‑paste mechanism, and that when a VLM is fine‑tuned for OCR, it largely preserves the same head identities while redistributing their functional and causal strengths.