← Back to all news
arXiv Machine Learning By Qinwu Xu, Yifan Jiang, Haoyu Ren

Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models

Read the original on arXiv Machine Learning →

arXiv:2605. 16409v3 Announce Type: replace-cross Abstract: Optical character recognition (OCR) and multilingual scene-text understanding remain challenging for multimodal large language models (MLLMs), particularly in real-world images containing small or degraded text, cluttered layouts, occlusion, handwriting, and complex typography.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computer Vision
Sep 22

All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

The paper introduces an all‑in‑one multilingual scene text recognizer called ScriptMoE, which uses a script‑aware mixture‑of‑experts architecture to handle 10 scripts and 229 languages. It is built on a new large‑scale synthetic dataset, TextMuSS‑10M, and evaluated on the TextMuSS‑Bench, achieving 82.06% accuracy—1.31% higher than the best baseline. When integrated into the PP‑OCRv5 pipeline, ScriptMoE raises the end‑to‑end multilingual F1 score from 65.71% to 80.89%, slightly surpassing the best vision‑language model while using far fewer parameters.

By Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen
llmscomputer-visionmultimodal
More like this →
arXiv Computer Vision
Aug 31

What Can Low Resource Languages Learn From Each Other?

The paper examines OCR adaptation for low‑resource languages, noting that fine‑tuning often hits a performance ceiling in data‑scarce settings. It identifies that lower layers of language‑specific models learn redundant features while higher layers capture script nuances, leading to a structural inefficiency. To address this, the authors propose PSMC, a framework that pre‑trains a base model, specializes it per language, merges the experts via task arithmetic, and co‑trains a unified multilingual backbone, achieving about a 2% improvement in Word Recognition Rate across 10 Indian scripts without adding parameters.

By Achyuth P, Kahaan Shah, Chetan Arora
llmsdiffusioncomputer-visionfine-tuningmultimodal
More like this →
arXiv Computation and Language
Aug 25

Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning

The paper introduces PM4Bench, a multimodal, multilingual, multi-task benchmark built on a strictly parallel 10‑language corpus, allowing fair cross‑lingual comparison of Large Vision‑Language Models (LVLMs). It also proposes a vision setting that embeds textual inputs directly into images to better mimic real deployment scenarios. Experiments show OCR performance drives cross‑lingual gaps, leading to an OCR‑centric GRPO training strategy that improves multilingual VQA and reduces disparities without costly supervision.

By Junyuan Gao, Jiahe Song, Jiang Wu, Runchuan Zhu, Guanlin Shen, Shasha Wang, Xingjian Wei, Haote Yang, Weijia Li, Bin Wang, Lijun Wu, Conghui He
llmsagentscomputer-visionreinforcement-learningmultimodalbenchmarks
More like this →