← Back to all news
arXiv AI

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

inkGround is a new benchmark for evaluating how well Vision‑Language Models can recover the fine‑grained stroke trajectories of cursive handwriting from camera‑captured page images. It contains 537 full‑page images with 18.7k words and 163k strokes, paired with pseudo‑trajectory references generated by an offline‑to‑online model and verified by humans. The benchmark casts the task as text‑conditioned point grounding, scoring models on trajectory fidelity, readability, and style similarity, and compares both open‑source and closed‑source VLMs, showing that closed‑source models like Gemini‑3.1‑Pro perform best.

Related stories

arXiv AI
Oct 16

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

InkSight is a system that converts offline handwritten notes captured in photos into digital ink, enabling seamless transition from pen-and-paper to editable, vectorized formats. Unlike prior methods that rely heavily on geometric image properties, InkSight integrates reading and writing priors, allowing it to learn from limited paired data. The model successfully derenders handwritten text in diverse photos and even simple sketches, with human evaluations showing 87% of outputs as valid tracings and 67% resembling human pen trajectories.

By Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai, Jesse Berent, Claudiu Musat
llmsmultimodal
More like this →
arXiv Computer Vision
Sep 3

Handwriting Trajectory Recovery via Autoregressive Ordered Stroke Instance Prediction

The paper presents a two‑stage method for recovering handwriting trajectories from static images. First, it predicts ordered stroke instances autoregressively, then reconstructs continuous motion within each stroke using direction‑related cues. Experiments on Chinese, English, and Tamil handwriting show that this ordered prediction outperforms post‑hoc ordering and baseline models, and that sampling density significantly impacts performance.

By En-Guang Wang, Yan-Ming Zhang, Fei Yin, Cheng-Lin Liu
More like this →
arXiv AI
Aug 20

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

OmniHandwritingOCR is a diagnostic benchmark designed to evaluate multimodal large language models (MLLMs) and OCR systems on handwritten text and mathematical expression recognition. It comprises 77.57K labeled images across six subtasks and twelve subsets, including a difficulty‑stratified multi‑line formula corpus that tests robustness to increasing structural complexity. The benchmark reveals that current systems perform poorly on complex multi‑line formulas, exhibit variable rankings across languages and formula settings, and sometimes hallucinate corrections that are not visually supported.

By Zinuo Guo, Min Zhang, Bo Jiang
llmscomputer-visionmultimodalbenchmarks
More like this →
arXiv AI
Aug 25

WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

WildHandBench is a new benchmark comprising 500 handwritten documents that span free text, tables, and formulas across four languages and nine real‑world scenarios. It introduces a Prior‑Driven Error (PDE) metric to assess whether mistakes stem from language priors rather than visual cues. In tests of 18 state‑of‑the‑art models, the best achieves only 71.85% accuracy, while humans reach 77.09%, and model errors are largely prior‑driven (63‑91%) compared to human errors (49%).

By Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, Shubin Han
benchmarks
More like this →