← Back to all news
arXiv Computer Vision By Niaz Mohaiman Abtahi, Reyana Shahid, Evan Maksud

Visual Sanad: Auditing Vision-Language Models for False Acceptance of Qur'anic Citations

Read the original on arXiv Computer Vision →

Visual Sanad is a new benchmark that tests vision‑language models (VLMs) for false acceptance of Qur’anic verse attributions across 2,400 images derived from 100 English Qur’an verses. The dataset includes four factual states per verse and six visual conditions, from plain text to decorative layouts, and all pairs were source‑checked and human‑audited. Experiments on 24 model‑prompt runs show a sharp split: proprietary API‑served VLMs vary from near‑perfect verification to moderate false acceptance, while recent open‑weight VLMs often default to VALID and show weak discrimination, though de‑biased prompting can reduce false acceptance by shifting thresholds toward INVALID. "whyItMatters":"The study reveals that fluent VLMs may treat displayed religious citations as self‑authenticating unless they possess strong citation‑verification ability, exposing a culturally grounded multimodal factuality failure."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computation and Language
4d ago

Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

The paper introduces BILAL-BENCH, a 240-question benchmark designed to evaluate open-ended Islamic-source fabrication in Bengali. It assesses models on Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial prompts, using a rubric that separates source errors from assertiveness. Seven open-weight models were tested, revealing that most confidently fabricate on 79.3–95.3% of source-production questions, with hadith provenance being the weakest area and no clear improvement with larger model size.

By Sabah Nushra, Md Kishor Morol
llmsbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 16

When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

The paper audits 12 vision‑language models on a 1,195‑image Arabic calligraphy benchmark, testing numeric, visual‑cue, and Diwani‑first prompts. Accuracy peaks at 60.9% for closed/API models and 47.2% for open‑weight models, but many models collapse to a single default label (often Thuluth) far above the majority‑class baseline. Prompting improves performance unevenly, with visual cues boosting some models by over 10 points while others suffer, and foregrounding Diwani often shifts but does not eliminate the default bias.

By Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid
llmsmultimodalbenchmarks
More like this →
arXiv AI
Oct 2

VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.

By Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
llmscomputer-visionreinforcement-learningmultimodalbenchmarks
More like this →