Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scie...
Doc‑CoB introduces a Chain‑of‑Boxes framework that enhances document understanding by progressively focusing on query‑relevant layout regions while preserving global context. It selects key layout boxes and then applies visual prompting for deeper analysis, supported by two new reasoning tasks and an automatic pipeline that generates 249k training samples with intermediate visual supervision. Experiments across seven benchmarks and four popular models demonstrate significant performance gains, underscoring the method’s effectiveness and broad applicability.
The paper introduces a new task called compositional layout understanding, focusing on interpreting complex, multi‑layer document and UI designs. It presents CoDeLayout, a VQA dataset of about 20,000 real‑world layouts annotated with compositional element pairs and design intent. The authors identify semantic drift and structural ambiguity as key challenges for vision‑language models and propose MASON, a post‑training approach that combines multimodal alignment and structural perception to improve performance, achieving 91.66% accuracy with only 30% of the training data.
arXiv:2607. 07179v1 Announce Type: cross Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents.
arXiv:2610.18238v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract p...
Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored.
DocAttriBench (DAB) is a large‑scale benchmark for fine‑grained, element‑level source attribution in Document Visual Question Answering (VQA). It introduces MAPPET, a Mask‑based Perplexity‑Derived Attribution method that uses document layout and language modeling to identify the most informative layout element for each answer. The benchmark contains 237k documents and 296k question‑answer pairs with element‑level grounding, and it evaluates multimodal LLMs on answer accuracy, attribution accuracy, and overall answer quality, revealing that even strong models often fail to localize supporting elements.
arXiv:2606. 04231v1 Announce Type: cross Abstract: Recent advances in multimodal retrieval-augmented generation (MM-RAG) have shifted toward minimal parsing, relying on page-level images for producing retriever embeddings and for answer generation.
The paper introduces the Structural Semantic Unit (SSU) and the Coverage, Overlap, Trespass, and Excess (COTe) score as a new framework for evaluating Document Layout Analysis (DLA) models. Unlike traditional metrics such as IoU, F1, or mAP, which are tailored to 2D projections of 3D space, COTe focuses on the semantic structure of printed media and is decomposable to reveal specific failure modes like breaching semantic boundaries or redundant parsing. Experiments on five common DLA models across three datasets show that COTe is more informative and robust—especially under granularity mismatches—than F1, and the authors provide an SSU-labelled dataset and a Python library to facilitate adoption.
arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.
DocAttriBench (DAB) is a large-scale benchmark that provides fine-grained, element-level source attribution for Document Visual Question Answering (VQA). It introduces MAPPET, a Mask-based Perplexity-Derived Attribution method that uses document layout and language modeling to identify the most informative layout element for each answer. The benchmark contains 237k documents and 296k question-answer pairs with grounding annotations, and it evaluates multimodal LLMs on answer accuracy, attribution accuracy, and overall answer quality.
The paper introduces a new task called compositional layout understanding, focusing on interpreting complex, multi-layer document and UI layouts that involve hierarchical relationships among visually entangled elements. It presents CoDeLayout, a VQA dataset of about 20,000 real-world layouts annotated with compositional element pairs and design intent, and identifies two main challenges for current vision‑language models: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter‑element relationships. To address these, the authors propose MASON, a post‑training paradigm that combines multimodal alignment and structural perception, achieving a 91.66% accuracy on CoDeLayout and outperforming full‑data direct fine‑tuning with only 30% of the training data.