ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation

ChartAttack is a framework that evaluates how multimodal large language models (MLLMs) can be misled by design misleaders to produce charts that cause incorrect interpretations. The authors also present AttackViz, a chart question‑answering dataset that labels effective misleaders and their induced wrong answers. Experiments show that ChartAttack can reduce MLLM QA accuracy by 17.2 points in‑domain and 11.9 points cross‑domain, and that fine‑tuning on AttackViz improves robustness to misleading charts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• fine-tuning
• multimodal