arXiv Computer Vision

Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

The paper introduces Invoice Haystack, a benchmark of 1,500 anonymized invoices and 200 question‑answer pairs that tests document retrieval and visual question answering under strong visual homogeneity. It shows that existing benchmarks suffer from embedding collapse, with Invoice Haystack’s mean pairwise cosine similarity at 0.73 versus 0.38 and 0.31 in DocHaystack and InfoHaystack. The authors propose VL‑RAG, a hybrid retrieval‑augmented generation framework that combines text and visual embeddings and a VLM‑based verification filter, achieving 60.0% Recall@1 on Invoice Haystack‑500 and improving performance on other benchmarks.

arXiv Computer Vision

Aug 27

VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval

VisDocAgentBench is a closed‑corpus benchmark that evaluates static versus agentic retrieval for visually rich documents, using 2,375 pages from 100 documents and 120 queries that span direct, one‑bridge, and two‑bridge evidence structures. The benchmark includes semantic, relational, and visual queries, full‑document review, and hard‑negative validation. Results show that a strong visual retriever performs well on direct items but poorly on two‑bridge items, while agents improve performance, especially when using visual retrieval and iterative search capabilities.

By Lexiang Hu, Yanzhao Zhang, Mingxin Li, Dingkun Long, Yikang Li, Fuwei Zhang, Yisen Wang, Zhouchen Lin

agents computer-vision nlp benchmarks

arXiv Computation and Language

Oct 27

BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering

arXiv:2604.22678v2 Announce Type: replace Abstract: A common approach to question answering with retrieval-augmented generation (RAG) is to concatenate documents into a single context and pass it to...

By Jinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne

llms rag nlp fine-tuning efficiency multimodal benchmarks

arXiv Computation and Language

Sep 15

TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Exams

arXiv:2609.13158v1 Announce Type: new Abstract: Large Vision--Language Models (LVLMs) are increasingly expected to perform visual question answering (VQA) over planar media. However, existing planar...

By Yongqi Yu, Yu Zhang

llms nlp benchmarks

arXiv AI

Jul 29

VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

arXiv:2607. 24748v1 Announce Type: cross Abstract: Visually-rich documents such as reports, slides, and manuals often distribute the evidence needed to answer a question across multiple pages, mixing text with layout cues, tables, charts, and figures.

By Seonok Kim

rag agents nlp multimodal benchmarks

arXiv Computation and Language

Oct 16

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

VELD-QA is a large-scale benchmark for visual evidence grounding and question answering in long documents, comprising 15,000 human-authored question-answer pairs across 5,859 PDFs with 372,445 pages. The dataset includes detailed annotations of supporting pages, visual evidence elements, and descriptions of cited visual content, enabling separate evaluation of retrieval, localization, and interpretation. Experiments on nine vision-language models and various retrieval-augmented systems reveal significant gaps in evidence localization and visual interpretation, with oracle accuracy peaking at 81.7% and enrichment of visual-element annotations narrowing performance gaps by 17–24%.

By Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yazhe Hu, Lan Guo, Yuying Wang, Mengqing Guo, Bhavesh Dnyaneshwar Gore, Shubham Pathak, Ramapriya Ranganath, Sandeep Sharma, Tao Sheng, Sujith Ravi, Dan Roth

llms rag agents nlp multimodal benchmarks

arXiv AI

Sep 3

ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering

ViSAR is a training‑free, adaptive‑k retrieval method for Visual Document Question Answering that operates directly in the embedding space to build a query‑conditioned page‑level similarity matrix. By dynamically selecting the number of pages to retrieve based on query relevance, ViSAR reduces Retrieval‑Augmented Generation latency by up to 58.7% while maintaining or improving answer accuracy across multiple encoders and Large Vision‑Language Models. The structure of the similarity matrix also correlates with answer accuracy, indicating potential for retrieval quality‑aware document understanding.

By Adrien Mialland, Marc Plantevit, Julien Gallois, C\'eline Robardet

llms rag nlp multimodal

arXiv Computer Vision

Sep 18

DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering

DocAttriBench (DAB) is a large‑scale benchmark for fine‑grained, element‑level source attribution in Document Visual Question Answering (VQA). It introduces MAPPET, a Mask‑based Perplexity‑Derived Attribution method that uses document layout and language modeling to identify the most informative layout element for each answer. The benchmark contains 237k documents and 296k question‑answer pairs with element‑level grounding, and it evaluates multimodal LLMs on answer accuracy, attribution accuracy, and overall answer quality, revealing that even strong models often fail to localize supporting elements.

By Luca De Grandis (University of Modena and Reggio Emilia, Modena, Italy), Silvia Cappelletti (University of Modena and Reggio Emilia, Modena, Italy), William Raccagni (University of Modena and Reggio Emilia, Modena, Italy, University of Pisa, Pisa, Italy), Marcella Cornia (University of Modena and Reggio Emilia, Modena, Italy), Lorenzo Baraldi (University of Modena and Reggio Emilia, Modena, Italy), Rita Cucchiara (University of Modena and Reggio Emilia, Modena, Italy)

llms nlp multimodal benchmarks

arXiv AI

Sep 10

Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment

arXiv:2609.08188v1 Announce Type: new Abstract: Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retriever...

By Zhan-Lun Chang, Dong-Jun Han, Seyyedali Hosseinalipour, Mung Chiang, Christopher G. Brinton

llms rag reinforcement-learning fine-tuning multimodal safety

arXiv AI

Oct 15

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

arXiv:2610.17057v1 Announce Type: new Abstract: Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This l...

By Simon Ott, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen

rag benchmarks

arXiv Computer Vision

Sep 1

Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation

arXiv:2608.30163v1 Announce Type: cross Abstract: Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, ex...

By Ruofan Hu, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Ke Lei, Tao Jin, Zhou Zhao

rag nlp reinforcement-learning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 9

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

arXiv:2607. 07179v1 Announce Type: cross Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents.

By Miguel Lopez-Duran, Elena Marrero, Julian Fierrez, Marta Robledo-Moreno, Ruben Vera-Rodriguez, Daniel DeAlcala, Aythami Morales, Ruben Tolosana, Oscar Delgado, Alvaro Ortigosa, Javier Ortega-Garcia

llms nlp fine-tuning multimodal

arXiv Computation and Language

Oct 15

Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

arXiv:2610.20367v1 Announce Type: new Abstract: Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layou...

By Diana Turmakhan, Aseem Srivastava, Vivek Gupta, Iryna Gurevych, Thamar Solorio

multimodal benchmarks

Related stories