← Back to all news
arXiv AI By Gwang Gook Lee, Kenan Emir Ak, Jay Mohta, Yan Xu, Dimitrios Dimitriadis

Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models

Read the original on arXiv AI →

Vision‑Language Models (VLMs) are increasingly replacing traditional OCR for document understanding, but this study shows they often rewrite imperfect text into more plausible forms, a flaw that clean‑text OCR benchmarks miss. The authors created FaithC4, a multilingual perturbation benchmark of 1,455 single‑page documents with scramble, random substitution, and visually similar substitution attacks, and evaluated 15 systems across general‑purpose VLMs, OCR‑specialized VLMs, and traditional OCR pipelines. Results reveal that general‑purpose VLMs suffer up to 6.9 WER points under perturbation, OCR‑specialized VLMs 0.1–3.4 points, and traditional OCR less than 0.8 points on English; probing Qwen3‑VL‑4B shows rewriting occurs only when a perturbed word’s final‑layer representation remains close to the original, with short words (4–6 characters) rewritten up to 10% of the time. whyItMatters":"The findings highlight a critical limitation of VLMs in document transcription, underscoring the need for robust evaluation benchmarks that capture rewriting behavior beyond clean‑text accuracy."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computer Vision
Sep 2

Can Scene Text Recognition Read Rare Compositions?

The paper reports that scene text recognition models, while achieving 89–97% accuracy on standard benchmarks, perform significantly worse on rare word–trigram combinations, with a 10–18 point drop in accuracy at the rare‑word/rare‑trigram corner across multiple languages and models. Scaling the vision backbone improves overall accuracy but does not alleviate this corner‑specific deficit. The authors identify the autoregressive decoder’s lexical prior as the root cause and show that architectural changes—specifically moving from autoregressive to CTC decoding—yield the largest improvement for these rare compositions.

By Genpei Zhang
benchmarks
More like this →
arXiv Computation and Language
Sep 30

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key-Value Extraction in Noisy Document Settings

The paper introduces a controlled benchmark for evaluating large language models (LLMs) on key‑value pair extraction from documents with varying levels of OCR noise. It tests 136 configurations across five instruction‑tuned open‑weight LLMs, three datasets, and four text‑quality conditions, using deterministic decoding to generate 17,688 document‑level inferences. The study finds that clean‑text performance does not reliably predict real‑world robustness, model rankings can reverse under noisy conditions, and few‑shot demonstrations do not always improve accuracy, highlighting reliability risks in OCR‑to‑LLM pipelines.

By Zahra Anvari
llmscomputer-visionmultimodalbenchmarks
More like this →
arXiv AI
3d ago

Entity-Aware Evaluation of OCR Systems for Historical Document Transcription

The paper introduces an entity‑aware evaluation framework for OCR systems applied to historical documents, focusing on how well names, dates, and identifiers are preserved. Using a benchmark of 175 Berrutti Archive documents and 5,209 manually annotated entity mentions, the authors compare sequence‑recognition OCR, OCR‑specialized vision‑language models, and general‑purpose vision‑language models. Their results show that OCR rankings vary with the evaluation criterion: Qwen 3.6 achieves the highest entity coverage, while some systems recover many entities when small character‑level errors are tolerated, indicating that aggregate transcription metrics alone do not capture meaningful information preservation.

By Camilo Mari\~no (UDELAR, CB), Diego Belzarena (UDELAR, CB), Marina Gardella (CB), Ignacio Ram\'irez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)
llmscomputer-visionmultimodalbenchmarks
More like this →