arXiv Computer Vision

Visual Sanad: Auditing Vision-Language Models for False Acceptance of Qur'anic Citations

Visual Sanad is a new benchmark that tests vision‑language models (VLMs) for false acceptance of Qur’anic verse attributions across 2,400 images derived from 100 English Qur’an verses. The dataset includes four factual states per verse and six visual conditions, from plain text to decorative layouts, and all pairs were source‑checked and human‑audited. Experiments on 24 model‑prompt runs show a sharp split: proprietary API‑served VLMs vary from near‑perfect verification to moderate false acceptance, while recent open‑weight VLMs often default to VALID and show weak discrimination, though de‑biased prompting can reduce false acceptance by shifting thresholds toward INVALID. "whyItMatters":"The study reveals that fluent VLMs may treat displayed religious citations as self‑authenticating unless they possess strong citation‑verification ability, exposing a culturally grounded multimodal factuality failure."

arXiv Computation and Language

4d ago

Do Open Language Models Invent Scripture? BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali

The paper introduces BILAL-BENCH, a 240-question benchmark designed to evaluate open-ended Islamic-source fabrication in Bengali. It assesses models on Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial prompts, using a rubric that separates source errors from assertiveness. Seven open-weight models were tested, revealing that most confidently fabricate on 79.3–95.3% of source-production questions, with hadith provenance being the weakest area and no clear improvement with larger model size.

By Sabah Nushra, Md Kishor Morol

llms benchmarks safety

arXiv AI

Aug 18

Catching Hallucinated Citations in Video-LLM Question Answering: A Self-Verification Pipeline and Verifier Ablation Study

arXiv:2608. 15574v1 Announce Type: cross Abstract: Video question answering systems built on vision-language models often produce timestamped claims with high confidence even when unsupported by the cited frame.

By Yogesh Kumar

llms rag nlp multimodal benchmarks safety

arXiv Computer Vision

Oct 1

Typographic Attack Against VLM-based AI-generated Image Detection

arXiv:2609.39662v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used for AI-generated image (AIGI) detection, providing natural-language explanations for authenticity j...

By Eunmin Lee, Jungwoo Kim, Jong-Seok Lee

llms multimodal

arXiv Computer Vision

Oct 16

When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition

The paper audits 12 vision‑language models on a 1,195‑image Arabic calligraphy benchmark, testing numeric, visual‑cue, and Diwani‑first prompts. Accuracy peaks at 60.9% for closed/API models and 47.2% for open‑weight models, but many models collapse to a single default label (often Thuluth) far above the majority‑class baseline. Prompting improves performance unevenly, with visual cues boosting some models by over 10 points while others suffer, and foregrounding Diwani often shifts but does not eliminate the default bias.

By Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid

llms multimodal benchmarks

arXiv AI

Oct 2

VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.

By Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen

llms computer-vision reinforcement-learning multimodal benchmarks

arXiv Computer Vision

Sep 30

Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning

arXiv:2609.37656v1 Announce Type: new Abstract: Large vision-language models (LVLMs) exhibit strong reasoning capabilities, yet the visual and textual evidence supporting the generated responses rema...

By Bowen Yuan, Danny Wang, Ruihong Qiu, Zijian Wang, Zi Huang

llms multimodal benchmarks

arXiv Computer Vision

Oct 9

False Claims, Credible Images: A Red-Teaming Benchmark for Commercial Image Generators

The paper introduces EpiReal-Bench, a benchmark of 10,000 false-claim prompts and corresponding images designed to test visual misinformation risks in commercial image generators. It also presents EpiReal-Attack, a black-box optimization framework that can bypass alignment safeguards, increasing the success rate of generating misleading images from 70% to 95%. Experiments on four commercial models show that these generators can produce credible visual evidence for false claims, highlighting a blind spot in current alignment methods.

By Zeyu Ye, Yanchun Li, Sibei He, Meng Xie, Hangtao Zhang, Xianlong Wang, Li Zeng, Jiahao Chen, Yichen Wang, Junhui Wang, Ziqi Zhou

llms multimodal benchmarks safety

arXiv AI

Jun 16

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

arXiv:2606. 16092v1 Announce Type: cross Abstract: Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements.

By Young Rok Jang, Hyesoo Kong, Kyunghwan An, Jae Sub Huh, Gyeonghun Kim, Stanley Jungkyu Choi

llms fine-tuning multimodal

arXiv Machine Learning

Jun 9

What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks

arXiv:2606. 09700v1 Announce Type: cross Abstract: Large language model (LLM)-powered content moderation systems have become a critical defense against harmful online content.

By Qin Yang, Lu Malloy, Joshua Lee, Xiaohan Chang, Meisam Mohammady, Doowon Kim, Yuan Hong

llms benchmarks safety

arXiv AI

Oct 19

Locating Failure in Multi-Page Visually Rich Document Understanding Systems: An Empirical Attribution

The paper introduces an empirical attribution framework to pinpoint failures in multi‑page visually rich document understanding systems. By applying controlled interventions in a unified document question‑answering pipeline, the authors analyze how evidence representation, selection, and usage affect performance. Experiments on MMLongBench‑Doc reveal that complementary modalities can mitigate conversion errors, that adding non‑gold pages can both correct and break answers, and that abstention strategies trade unsupported answers for false refusals, with varying effects across different backbones.

By Lewei Xu, Yihao Ding, Zihan Xu, Daniel Yitian Su, Daochang Liu, Siwen Luo, Yifan Peng, Wei Liu

nlp

arXiv AI

Jul 28

AI-generated Images Challenge Visual Trust in High-risk Scenarios

arXiv:2607. 22745v1 Announce Type: cross Abstract: Rapid advances in image generation are eroding the evidentiary value of visual content in settings where authenticity can affect public safety and personal reputation.

By Yi-Zhi Wang, Yichen Xiao, Linan Yue, Weibo Gao, Yichao Du, Pengfei Fang, Shimin Di, Min-Ling Zhang

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 3

SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation

arXiv:2606. 03348v1 Announce Type: cross Abstract: Recent generative models can now produce visual artifacts with realistic embedded text and layouts, creating a new misinformation threat: synthetic credibility.

By Junxiao Yang, Minghao Zhang, Xiaoce Wang, Haoran Liu, Shiyao Cui, Hongning Wang, Minlie Huang

benchmarks

Related stories