arXiv AI

Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks

The paper proposes universal, tool‑based defenses for large language model agents that use external tools, addressing four types of adversarial attacks: direct and indirect prompt injection, memory poisoning, and backdoor attacks. Two main defenses are introduced: Attacker Tool Filtering, which uses anomaly detection to remove suspicious tools, and Normal Tool Recalling, which restores the agent’s original toolset before planning. The authors also add prompt‑based defenses such as Chain‑of‑Thought prompting and self‑reflection, and demonstrate that these methods dramatically lower attack success rates—often to 0%—across multiple open‑source and proprietary LLMs while maintaining or improving task performance.

arXiv AI

Oct 16

INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

INFUSE is a new benchmark that tests indirect prompt injection (IPI) in large‑language‑model agents performing long‑horizon workflows. It contains 40 realistic tasks and 56 attack scenarios across five threat categories, with attacks injected through realistic interaction surfaces rather than just initial prompts. The study evaluated 19 LLMs, generating over 41,000 adversarial trajectories and found mean attack success rates of 19.6% (LLM evaluation) and 15.4% (heuristic verification), while prompt‑level hardening reduced success from 58.2% to 14.3%.

By Thuy Dung Nguyen, Jingyi Guo, Rana Seda Dubauskas, Sreepriya Damuluru, Taylor T. Johnson, Kevin Leach

llms agents multimodal benchmarks safety

arXiv AI

Sep 15

ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents

arXiv:2609.14987v1 Announce Type: cross Abstract: Large language model (LLM) agents interact with external environments through tool invocation, but tool outputs can also expose them to indirect prom...

By Bingzheng Wang, Xiaoyan Gu, Wentao Wang, Xingyou Yang, Hongcheng Li, Rong Yin

llms agents benchmarks

arXiv AI

Jun 16

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

arXiv:2606. 15441v1 Announce Type: cross Abstract: Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.

By Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan

llms rag agents reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 21

STAC: When Innocent Tools Form Dangerous Chains for LLM Agents

arXiv:2509. 25624v3 Announce Type: replace-cross Abstract: As LLMs advance into autonomous agents with tool-use capabilities, they introduce security challenges that extend beyond traditional content-based LLM safety concerns.

By Jing-Jing Li, Jianfeng He, Chao Shang, Devang Kulshreshtha, Xun Xian, Yi Zhang, Hang Su, Sandesh Swamy, Yanjun Qi

llms agents multimodal benchmarks safety

arXiv AI

Jun 10

Assessing Automated Prompt Injection Attacks in Agentic Environments

arXiv:2606. 10525v1 Announce Type: cross Abstract: Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings.

By David Hofer, Edoardo Debenedetti, Florian Tram\`er

llms agents safety

Hugging Face Trending Papers

Jun 13

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution. Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.

llms rag agents reinforcement-learning multimodal benchmarks safety

arXiv AI

Aug 5

Your Agentic LLMs Secretly Encode Latent Signals of Indirect Prompt-Injection Exposure

arXiv:2608. 02657v1 Announce Type: cross Abstract: Agentic LLMs are vulnerable to indirect prompt injection (IPI) attacks, e.

By Jianshuo Dong, Yiming Liu, Maosen Zhang, Nan Deng, Xu Peng, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu

llms agents

arXiv AI

Jun 2

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

arXiv:2603. 24511v2 Announce Type: replace-cross Abstract: We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations.

By Alexander Panfilov, Peter Romov, Igor Shilov, Yves-Alexandre de Montjoye, Jonas Geiping, Maksym Andriushchenko

llms agents multimodal benchmarks safety

arXiv Computation and Language

Oct 27

Dummy Backdoor as a Defense: Removing Unknown Backdoors via Shared Internal Mechanisms for Generative LLMs

arXiv:2606.11648v2 Announce Type: replace-cross Abstract: Backdoor attacks pose a serious threat to the safety and reliability of Large Language Models (LLMs), as they cause models to behave normally...

By Kazuki Iwahana, Masaru Matsubayashi, Takuma Koyama, Toshiki Shibahara, Kenichiro Ominato, Akira Ito

llms fine-tuning safety

arXiv AI

Jun 16

AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

arXiv:2606. 15057v1 Announce Type: cross Abstract: Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.

By Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

llms agents multimodal benchmarks

arXiv AI

Jun 15

Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH

arXiv:2606. 13994v1 Announce Type: cross Abstract: LLM-based Agents are becoming increasingly capable and widely deployed, creating growing incentives for adversarial misuse in the real-world.

By Vikhyath Kothamasu, Virginia Smith, Chhavi Yadav

llms agents benchmarks safety

arXiv AI

Aug 26

What Guides the Agent? Adjudicating Unauthorized Behavior via Localizing Behavior-Guiding Instructions

The paper introduces Attnlocate, a runtime framework that localizes behavior‑guiding instructions within the attention matrix of large language model agents. By treating this localization as an object detection task, Attnlocate uses a multi‑head, multi‑layer attention aggregation scheme and a 1‑D U‑Net to identify spans that influence tool‑calling decisions. The system then adjudicates potential malicious invocations based on the authority of the source, achieving high detection metrics across diverse LLM families and demonstrating transferability to unseen models.

By Yichao Gao, Yumo Zhang, Yunhao Yao, Haohua Du, Puhan Luo, Ruiqi Li, Zhiqiang Wang

llms agents computer-vision

Related stories