← Back to all news
arXiv Computer Vision By Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid

When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

Read the original on arXiv Computer Vision →

The paper audits 12 vision‑language models on a 1,195‑image Arabic calligraphy benchmark, testing numeric, visual‑cue, and Diwani‑first prompts. Accuracy peaks at 60.9% for closed/API models and 47.2% for open‑weight models, but many models collapse to a single default label (often Thuluth) far above the majority‑class baseline. Prompting improves performance unevenly, with visual cues boosting some models by over 10 points while others suffer, and foregrounding Diwani often shifts but does not eliminate the default bias.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Sep 16

Same Answer, Different Representations: Hidden instability in VLMs

arXiv:2602.06652v2 Announce Type: replace Abstract: The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions r...

By Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema, Wai-Chung Kwan, Fazl Barez, Maria Sofia Bucarelli, Fabrizio Silvestri, Pasquale Minervini
llmsragmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 9

Multimodal Large Language Models as Image Classifiers

The paper investigates how evaluation protocols and ground‑truth quality affect the classification performance of Multimodal Large Language Models (MLLMs). It identifies and corrects key issues such as discarded out‑of‑list outputs, inflated distractor choices, and poor open‑world mapping, and shows that design choices like batch size, image ordering, and text encoder selection significantly influence accuracy. Using a multilabel reannotation of 625 ImageNet‑1k classes (ReGT), the study finds that corrected labels can boost MLLM performance by up to 10.8%, narrowing the gap with supervised models, and demonstrates that MLLMs can assist human annotators in about half of difficult cases.

By Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas
llmsmultimodal
More like this →