arXiv AI

Chart Deception in Vision-Language Models: From Vulnerability to Mitigation

arXiv:2607. 22600v1 Announce Type: new Abstract: Information visualizations are widely used to communicate patterns, trends, and outliers, yet deceptive design choices-such as truncated or inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings-can systematically alter interpretation while preserving the underlying data.

arXiv AI
Jul 15

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

arXiv:2603. 28583v2 Announce Type: replace-cross Abstract: Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations.

By Yanjie Zhang, Yafei Li, Rui Sheng, Zixin Chen, Yanna Lin, Huamin Qu, Lei Chen, Yushi Sun
arXiv Computation and Language
Sep 23

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

The paper introduces ChartBias, a benchmark of 820 real-world charts covering six social attributes, designed to audit bias in vision‑language models (VLMs) that interpret charts. Across 12 VLMs, the study identifies three failure modes—narrative shift, group hallucination, and preference polarity—where models produce different or misleading narratives when the referenced social group changes. A multi‑agent mitigation framework is proposed, separating evidence extraction from group‑conditioned generation and using a counterfactual judge, which reduces narrative shift while maintaining chart‑grounded reasoning.

By Mizanur Rahman, Huan Wu, Arash Asgari, Enamul Hoque Prince, Laleh Seyyed-Kalantari
arXiv Computation and Language
Sep 3

ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation

ChartAttack is a framework that evaluates how multimodal large language models (MLLMs) can be misled by design misleaders to produce charts that cause incorrect interpretations. The authors also present AttackViz, a chart question‑answering dataset that labels effective misleaders and their induced wrong answers. Experiments show that ChartAttack can reduce MLLM QA accuracy by 17.2 points in‑domain and 11.9 points cross‑domain, and that fine‑tuning on AttackViz improves robustness to misleading charts.

By Jesus-German Ortiz-Barajas, Jonathan Tonglet, Vivek Gupta, Iryna Gurevych
arXiv AI
Aug 26

Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes

The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.

By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding
arXiv Computer Vision
Sep 24

ASAP: Visual Analytics for Identifying and Analyzing Image Patterns in AI-generated Images

ASAP is an interactive visualization system that helps users identify and analyze deceptive patterns in AI‑generated images. It uses a CLIP‑adapted image encoder to produce interpretable representations and generates masks that highlight influential pixel regions, enabling influence measurement of key deceptive features. The system integrates these techniques into a dashboard for quantifying authenticity‑indicative patterns across collections of authentic and AI‑generated images, supporting comparative analysis of different generative models such as GANs and diffusion models, and its effectiveness is demonstrated through a user study and benchmark applications.

By Jinbin Huang, Yuki Ueno, Chen Chen, Aditi Mishra, Bum Chul Kwon, Zhicheng Liu, Chris Bryan