arXiv AI

CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering

arXiv AI
Nov 6

Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks

The paper introduces a new benchmark for indirect prompt injection (IPI) that focuses on attacks which tamper with decisions already in progress rather than redirecting the task. In 90 validated cases across three tool‑use environments, attackers target up to six aspects of an agent’s decision‑making—objective, decision rules, available actions, expected consequences, observation reading, or state beliefs—while keeping the original task intact. The study finds that many defenses reduce objective‑targeting attacks but leave observation and state attacks largely effective, highlighting gaps in current mitigation strategies.

By Rongting Zhang, Martin Bertran, Sandeep Avula, Chia-Jung Lee, Vanessa Murdock
arXiv AI
Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney
arXiv AI
Jun 26

Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

arXiv:2606. 26479v1 Announce Type: cross Abstract: Recent work (2024 to 2026) has converged on a strategy for defending tool-using LLM agents against indirect prompt injection: rather than training the model to refuse malicious instructions, enforce security outside the model with a deterministic policy that mediates the agent's actions.

By Praneeth Narisetty, Shiva Nagendra Babu Kore, Uday Kumar Reddy Kattamanchi, Jayaram Kumarapu
arXiv AI
Oct 28

AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification

AgentSentry is an inference‑time defense for tool‑augmented large language model agents that mitigates indirect prompt injection (IPI) attacks. It uses temporal causal diagnostics by performing counterfactual re‑executions at tool‑return boundaries to detect mediator‑sensitive proposals, purifies implicated content, and resumes execution from a sanitized state. Evaluations on AgentDojo show zero successful attacks (ASR = 0%) and an average Utility Under Attack of 76.10%, outperforming baselines while maintaining low false‑positive rates.

By Tian Zhang, Yiwei Xu, Juan Wang, Keyan Guo, Xiaoyang Xu, Bowen Xiao, Quanlong Guan, Jinlin Fan, Jiawei Liu, Zhiquan Liu, Hongxin Hu
arXiv AI
Sep 16

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

The paper proposes universal, tool‑based defenses for large language model agents that use external tools, addressing four types of adversarial attacks: direct and indirect prompt injection, memory poisoning, and backdoor attacks. Two main defenses are introduced: Attacker Tool Filtering, which uses anomaly detection to remove suspicious tools, and Normal Tool Recalling, which restores the agent’s original toolset before planning. The authors also add prompt‑based defenses such as Chain‑of‑Thought prompting and self‑reflection, and demonstrate that these methods dramatically lower attack success rates—often to 0%—across multiple open‑source and proprietary LLMs while maintaining or improving task performance.

By Xiaoyan Li, Yunli Wang
arXiv Computation and Language
Oct 28

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

arXiv:2610. 37657v1 Announce Type: cross Abstract: We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates.

By Igor Shilov, Xiaoxue Yang, Alexander Panfilov, Evgenii Kortukov, Matthieu Meeus, Peter Romov, Wojciech Samek, Jonas Geiping, Yves-Alexandre de Montjoye