arXiv AI

Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents

The paper introduces covert indirect prompt injection (IPI) attacks on tool‑using large language model agents, distinguishing between covert and overt successes. It defines new metrics—Covert Success Rate (CSR) and Overt Success Rate (OSR)—to capture whether users notice the injection. The authors propose ICoA, an attack that steers agents back to the user’s task after executing the injection, achieving higher CSR than existing methods on four target models.

arXiv AI
Aug 13

Backdoor Decontamination Dynamics in LLM Agents

arXiv:2608. 11295v1 Announce Type: cross Abstract: Open-weight LLM agents are vulnerable to backdoors installed during fine-tuning, which may be undetectable if the trigger conditions are never met during testing.

By Gabriel Huang, Abhay Puri, L\'eo Boisvert, Alexandre Drouin, Perouz Taslakian, Spandana Gella, Christopher Pal
arXiv AI
2d ago

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

The paper proposes universal, tool‑based defenses for large language model agents that use external tools, addressing four types of adversarial attacks: direct and indirect prompt injection, memory poisoning, and backdoor attacks. Two main defenses are introduced: Attacker Tool Filtering, which uses anomaly detection to remove suspicious tools, and Normal Tool Recalling, which restores the agent’s original toolset before planning. The authors also add prompt‑based defenses such as Chain‑of‑Thought prompting and self‑reflection, and demonstrate that these methods dramatically lower attack success rates—often to 0%—across multiple open‑source and proprietary LLMs while maintaining or improving task performance.

By Xiaoyan Li, Yunli Wang
arXiv AI
Sep 7

Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection

Repeat-After-Me is a black-box adaptive visual prompt injection technique that can reveal personally identifiable information or trigger malicious tool calls in both open-weight and commercial vision‑language models, achieving attack success rates above 80% on Qwen3.6‑27B and 47% on GPT‑5.5. The method works even when the benign user prompt is unrelated to the injected task and does not explicitly authorize it, and it retains significant effectiveness when transferred across models or optimized on surrogate systems. In a real‑world OpenClaw Discord deployment, a minimally injected image can overwrite TOOLS.md, enabling remote code execution and secret exfiltration.

By Sizhe Chen, Yu-Lin Tsai, Ivan Evtimov, Kamalika Chaudhuri, Raluca Ada Popa, David Wagner, Arman Zharmagambetov
arXiv Machine Learning
Jul 30

ToxScreen: Detecting Whether an LLM Has Been Poisoned

arXiv:2607. 26849v1 Announce Type: cross Abstract: As large language models (LLMs) are deployed in high-stakes domains, adversaries may poison training data to implant backdoors: hidden triggers that covertly manipulate model behavior at inference time.

By Anthony Hughes, Nicole Xing, Collin Francel, Andy Kim, Andrew Draganov