The paper introduces MisVisAgentBench, a paired benchmark with 280 task instances that compare clean versus misleading chart evidence for web agents. It evaluates how deceptive visualizations affect agent execution, providing a diagnostic framework to separate misleading-induced failures from baseline difficulty. Results show that misleading charts reduce task success by over 34 percentage points and tend to redirect agents toward targeted misleading actions.
arXiv:2607. 22600v1 Announce Type: new Abstract: Information visualizations are widely used to communicate patterns, trends, and outliers, yet deceptive design choices-such as truncated or inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings-can systematically alter interpretation while preserving the underlying data.
arXiv:2608.24535v1 Announce Type: new Abstract: Data visualizations are widely used for communicating information, but they are also vulnerable to intentional manipulations that induce misleading int...
arXiv:2603. 28583v2 Announce Type: replace-cross Abstract: Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations.
arXiv:2609.01383v1 Announce Type: new Abstract: Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic...
arXiv:2608. 08392v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents that interact with the web through browsers.
Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interacti...
ScamWorld is an interactive environment that recreates reported scam websites across various scenarios, fraud mechanisms, and languages, allowing researchers to observe fraud indicators before a web agent’s final action. The platform enables controlled, repeatable evaluation of web agents, which are increasingly used to browse, shop, and manage accounts on the open web. A study of six leading LLM-powered agents revealed deception rates between 11% and 35%, with agents more likely to proceed with payments than to provide credentials or wallet information, and some agents continuing after recognizing suspicious conditions.
arXiv:2610.09240v1 Announce Type: cross Abstract: Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actio...
ChartAttack is a framework that evaluates how multimodal large language models (MLLMs) can be misled by design misleaders to produce charts that cause incorrect interpretations. The authors also present AttackViz, a chart question‑answering dataset that labels effective misleaders and their induced wrong answers. Experiments show that ChartAttack can reduce MLLM QA accuracy by 17.2 points in‑domain and 11.9 points cross‑domain, and that fine‑tuning on AttackViz improves robustness to misleading charts.
The paper introduces a unified framework for monitoring misuse of large language model agents in multi‑actor settings, focusing on two attack types: decomposition attacks and prompt injection attacks. It proposes tracking the agent’s externalised actions and determining whether the first detection of harm falls within a defined harm window. Using this formalism, the authors build a benchmark of about 6,200 conversation transcripts and evaluate 17 monitor configurations, finding that action‑framed monitors perform well on both threats while content‑framed monitors struggle with injection attacks, and highlighting limitations of traditional position‑blind metrics.
arXiv:2602. 15278v2 Announce Type: replace-cross Abstract: The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs).
The paper introduces EpiReal-Bench, a benchmark of 10,000 false-claim prompts and corresponding images designed to test visual misinformation risks in commercial image generators. It also presents EpiReal-Attack, a black-box optimization framework that can bypass alignment safeguards, increasing the success rate of generating misleading images from 70% to 95%. Experiments on four commercial models show that these generators can produce credible visual evidence for false claims, highlighting a blind spot in current alignment methods.