← Back to all news
arXiv Computation and Language By Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma

ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction

Read the original on arXiv Computation and Language →

ShredBench is a new benchmark for evaluating multimodal large language models (MLLMs) on the task of reconstructing documents from shredded fragments. It uses an automated pipeline that generates fragmented documents from Markdown, covering English, Chinese, code, and table scenarios with varying fragmentation levels. Experiments show that while MLLMs perform well on intact documents, their performance drops sharply when documents are shredded, revealing a gap in fine‑grained cross‑modal reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
1d ago

AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing

AmbiHTMLBench is a new benchmark that evaluates how well models can convert real-world documents into executable HTML while preserving content, organization, and visual appearance. It contains 1,115 pages with structured annotations and uses both objective metrics (table fidelity, text error, layout consistency) and a reference-anchored subjective protocol (LLM judges aligned with human preferences). Results show that different systems excel in objective versus subjective measures, and fine‑tuning a small expert model improves performance, highlighting the benchmark’s role as a diagnostic tool for advanced document parsing.

By Boyu Niu, Yingjie Xiong, Shurui Li, Linhan Pang, Yuntong Zhang, Han Wang, Hao Wu, Kai Zuo, Xu Tang, Xuanhe Zhou, Fan Wu
llmsfine-tuningmultimodalbenchmarks
More like this →
arXiv Computation and Language
Oct 28

DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

DocXEval is the first comprehensive framework that adapts existing explainable AI methods and evaluation metrics to multimodal document understanding, covering text, layout, and image modalities. The authors systematically evaluate 13 XAI methods across 3 tasks, 8 datasets, 4 models, and 17 metrics, finding that Occlusion consistently performs best in faithfulness and interpretability. The study also provides meta‑evaluation insights, diagnostic analyses of modality contributions, spatial attribution patterns, and potential dataset biases, offering a benchmark and practical guide for future research.

By Saifullah Saifullah, David Dembinsky, Stefan Agne, Andreas Dengel, Sheraz Ahmed
llmsmultimodalbenchmarks
More like this →