Multimodal LLMs can see a document, but they often can't read it reliably. Small text, tables, visual cues, and topological elements still trip them up under direct visual inference, even when the pag...
The paper investigates the gap between retrieval and reading in document vision‑language models, showing that even when the correct page is retrieved, the model often fails to use the text on that page. By comparing answers derived from page images alone versus images plus extracted OCR text, the authors demonstrate that adding OCR text can improve strict accuracy by 13 to 16 points on their FoveDoc‑Bench benchmark. The study also reveals that OCR benefits textual evidence but not charts or figures, and that its advantage diminishes as retrieval quality worsens.
By Qingtao Xia, Siyao Cheng, Jiahua Bao, Jiaxing Du, Jie Liu
The paper investigates how general vision‑language models (VLMs) develop specialized optical character recognition (OCR) capabilities. By applying a causal intervention protocol, the authors identify sparse, stable OCR‑head sets in several VLMs and show that these heads largely overlap with textual retrieval/copy heads found in general VLMs. The study concludes that full‑sequence OCR functions as a dense multimodal copy‑and‑paste mechanism, and that when a VLM is fine‑tuned for OCR, it largely preserves the same head identities while redistributing their functional and causal strengths.
By Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui
Vision‑Language Models (VLMs) are increasingly replacing traditional OCR for document understanding, but this study shows they often rewrite imperfect text into more plausible forms, a flaw that clean‑text OCR benchmarks miss. The authors created FaithC4, a multilingual perturbation benchmark of 1,455 single‑page documents with scramble, random substitution, and visually similar substitution attacks, and evaluated 15 systems across general‑purpose VLMs, OCR‑specialized VLMs, and traditional OCR pipelines. Results reveal that general‑purpose VLMs suffer up to 6.9 WER points under perturbation, OCR‑specialized VLMs 0.1–3.4 points, and traditional OCR less than 0.8 points on English; probing Qwen3‑VL‑4B shows rewriting occurs only when a perturbed word’s final‑layer representation remains close to the original, with short words (4–6 characters) rewritten up to 10% of the time.
whyItMatters":"The findings highlight a critical limitation of VLMs in document transcription, underscoring the need for robust evaluation benchmarks that capture rewriting behavior beyond clean‑text accuracy."
By Gwang Gook Lee, Kenan Emir Ak, Jay Mohta, Yan Xu, Dimitrios Dimitriadis
arXiv:2609.38368v1 Announce Type: new
Abstract: Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA bench...
By L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna, Satyajit Tourani, Ashshak Sharifdeen, Fida Mohammad Thoker, Bernard Ghanem, Muhammad Haris Khan
LatentPress compresses conversational histories and long documents into continuous memory tokens that a frozen decoder can read directly, eliminating the need for text reconstruction at inference. The method achieves 4–16× compression with only a small adapter (0.1% of the decoder’s parameters) and outperforms text summaries and OCR-based compression on LongMemEval and LongBench-QA benchmarks. Writing and reading are significantly faster than traditional text summarization or OCR reconstruction, demonstrating a practical machine-facing context interface beyond text and vision.
By Zhengze Zhou, Hejian Sang