ShredBench is a new benchmark for evaluating multimodal large language models (MLLMs) on the task of reconstructing documents from shredded fragments. It uses an automated pipeline that generates fragmented documents from Markdown, covering English, Chinese, code, and table scenarios with varying fragmentation levels. Experiments show that while MLLMs perform well on intact documents, their performance drops sharply when documents are shredded, revealing a gap in fine‑grained cross‑modal reasoning.
arXiv:2608.30163v1 Announce Type: cross Abstract: Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, ex...
AmbiHTMLBench is a new benchmark that evaluates how well models can convert real-world documents into executable HTML while preserving content, organization, and visual appearance. It contains 1,115 pages with structured annotations and uses both objective metrics (table fidelity, text error, layout consistency) and a reference-anchored subjective protocol (LLM judges aligned with human preferences). Results show that different systems excel in objective versus subjective measures, and fine‑tuning a small expert model improves performance, highlighting the benchmark’s role as a diagnostic tool for advanced document parsing.
DocXEval is the first comprehensive framework that adapts existing explainable AI methods and evaluation metrics to multimodal document understanding, covering text, layout, and image modalities. The authors systematically evaluate 13 XAI methods across 3 tasks, 8 datasets, 4 models, and 17 metrics, finding that Occlusion consistently performs best in faithfulness and interpretability. The study also provides meta‑evaluation insights, diagnostic analyses of modality contributions, spatial attribution patterns, and potential dataset biases, offering a benchmark and practical guide for future research.
arXiv:2610.18238v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract p...
arXiv:2608.29037v1 Announce Type: cross Abstract: Real-world degradations such as blur, shadow, distortion, and moire patterns severely impair the document question-answering capabilities of Multimod...
arXiv:2607. 10400v1 Announce Type: cross Abstract: Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc.
arXiv:2610.20335v1 Announce Type: new Abstract: Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction,...
Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scie...
The paper introduces a controlled benchmark for evaluating large language models (LLMs) on key‑value pair extraction from documents with varying levels of OCR noise. It tests 136 configurations across five instruction‑tuned open‑weight LLMs, three datasets, and four text‑quality conditions, using deterministic decoding to generate 17,688 document‑level inferences. The study finds that clean‑text performance does not reliably predict real‑world robustness, model rankings can reverse under noisy conditions, and few‑shot demonstrations do not always improve accuracy, highlighting reliability risks in OCR‑to‑LLM pipelines.
arXiv:2606. 09578v1 Announce Type: new Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored.
arXiv:2607. 07179v1 Announce Type: cross Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents.
arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.