HakushoBench is a Japanese chart and table visual question answering benchmark created from 33 governmental white papers, comprising 2,053 images across more than ten types. The dataset includes manually annotated and independently verified QA pairs that evaluate holistic understanding of charts and tables rather than just local visual cues. Experiments show that HakushoBench is significantly harder than existing Japanese benchmarks, with sub‑10B open‑weight models achieving at most 58.6% accuracy and even large models like Qwen3.5-397B-A17B lagging behind Gemini‑3‑Pro by 8.1 points.
Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering (TabVQA). While recent approaches predominantly rel...
arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.
arXiv:2609.17458v1 Announce Type: cross Abstract: Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering...
arXiv:2607. 09068v1 Announce Type: cross Abstract: Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning.
arXiv:2603. 27223v2 Announce Type: replace-cross Abstract: We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union.
arXiv:2608.28635v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have advanced document understanding, visual question answering, and text extraction. However, their...
BEAR-Bench is a bilingual benchmark for multimodal large language models, featuring 1,000 human‑annotated questions derived from text‑rich business and scientific documents in English and Russian. It evaluates 16 MLLMs, including Gemini 3.1 Pro and Qwen3.5‑397B, revealing significant performance gaps even for the strongest systems. The benchmark also serves to compare hallucination‑detection methods by analyzing model failures on these complex documents.
SCAFFOLD is a large-scale structured dataset of computer science research figures, each paired with captions, context, questions, answers, and chain-of-thought reasoning traces. It contains 157,387 figure–question pairs from 3,058 arXiv papers, with subsets of 36,797 and 12,000 pairs for medium and small-scale use. The dataset was created using layout detection, PDF parsing, and AI-assisted question generation, and was used to benchmark a vision‑language model (Qwen2.5‑VL‑3B‑Instruct).
arXiv:2606. 09578v1 Announce Type: new Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored.
arXiv:2607. 05614v1 Announce Type: cross Abstract: Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications.
arXiv:2609.08657v1 Announce Type: cross Abstract: Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This str...
The paper introduces CompareBench, a new benchmark suite for evaluating visual comparison reasoning in vision‑language models. It includes TallyBench for object counting, OmniCaps for captioning and tagging, and a 1,200‑question CompareBench that tests quantity, geometric, spatial, and temporal comparisons. Experiments on nine closed‑source models show strong overall performance but persistent weaknesses in counting, spatial reasoning, geometric comparison, and temporal ordering, highlighting visual comparison as a systematic challenge for current VLMs.