← Back to all news
arXiv AI By Zhihao Liu, Hongyu Sun, Zhiyuan Fu, Xiaonan Duan, Jice Wang, Shangru Zhao, Weizhi Meng, Wuxin Yang, Yangfan Zhou, Yuqing Zhang

AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

Read the original on arXiv AI →

The paper introduces an end‑to‑end framework for testing image‑triggered command injection on computer‑use agents (CUAs). It demonstrates that a local visual patch can cause verifiable environmental effects through the entire pipeline—from screenshot input, through vision‑language‑model generation, action parsing, to execution—by training and deploying patches on GitHub Pages and a CSDN clone. Across five open‑source or publicly available GUI‑agent or VLM backends, the study reports 84.5% T‑ASR, 47.0% TAPR, and 20.3% E2E‑ASR success rates, with trajectory analysis revealing that some attacks first execute malicious terminal commands before continuing the original task.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 7

Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection

Repeat-After-Me is a black-box adaptive visual prompt injection technique that can reveal personally identifiable information or trigger malicious tool calls in both open-weight and commercial vision‑language models, achieving attack success rates above 80% on Qwen3.6‑27B and 47% on GPT‑5.5. The method works even when the benign user prompt is unrelated to the injected task and does not explicitly authorize it, and it retains significant effectiveness when transferred across models or optimized on surrogate systems. In a real‑world OpenClaw Discord deployment, a minimally injected image can overwrite TOOLS.md, enabling remote code execution and secret exfiltration.

By Sizhe Chen, Yu-Lin Tsai, Ivan Evtimov, Kamalika Chaudhuri, Raluca Ada Popa, David Wagner, Arman Zharmagambetov
llmsagentsmultimodal
More like this →