arXiv Machine Learning

CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement

The paper introduces CoER, a framework that defends language‑model agents against adaptive indirect prompt injection (IPI) by employing attacker‑defender co‑evolution and refinement. CoER models IPI as a general‑sum Markov game, uses Co‑PPO to maintain historical opponent populations, and fine‑tunes defenders only on verified safe demonstrations. In experiments across seven domains, CoER cuts attack success from 38.5% to 0.2% while boosting task utility from 63.2% to 76.3%.

arXiv Machine Learning

Sep 10

CoRL: Co-Evolutionary Reinforcement Learning for Adaptive Indirect Prompt-Injection Attacks and Defenses

The paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to defend against adaptive indirect prompt-injection attacks on tool-augmented language agents. CoRL operates in three stages—attacker fine‑tuning, bilateral Co‑PPO training, and defender fine‑tuning—using verifier‑grounded repairs to adapt to changing attack strategies. Experiments on 1,514 executions show that CoRL reduces attack success rates to 0% while improving task utility, demonstrating its effectiveness against adaptive adversaries.

By Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu

agents reinforcement-learning fine-tuning multimodal benchmarks safety

Hugging Face Trending Papers

Jun 13

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution. Existing defenses report near-zero attack success rate on static benchmarks, yet recent adaptive evaluations show that these results collapse once the attacker is allowed to optimize against the deployed defense.

llms rag agents reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 16

Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment

arXiv:2606. 15441v1 Announce Type: cross Abstract: Indirect prompt injection attacks hijack LLM-based agents by embedding malicious instructions in third-party data that the agent retrieves during task execution.

By Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan

llms rag agents reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jul 24

PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

arXiv:2603. 13026v2 Announce Type: replace Abstract: Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents.

By Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia

llms agents reinforcement-learning efficiency benchmarks safety

arXiv AI

Jun 16

AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

arXiv:2606. 15057v1 Announce Type: cross Abstract: Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.

By Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

llms agents multimodal benchmarks

arXiv AI

Jun 11

Learning to Inject: Automated Prompt Injection via Reinforcement Learning

arXiv:2602. 05746v2 Announce Type: replace-cross Abstract: Prompt injection is a critical vulnerability in LLM agents, yet the strongest methods still rely on human red-teamers and hand-crafted prompts.

By Xin Chen, Jie Zhang, Florian Tram\`er

llms agents reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 7

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

arXiv:2506. 07468v4 Announce Type: replace Abstract: Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.

By Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 30

AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents

arXiv:2607. 26998v1 Announce Type: cross Abstract: Large language model (LLM) agents automate penetration testing through an observation-action loop, selecting actions based on observations returned by tools.

By Ruoyu Wang, Heng Zhao, Renjie Wu, Mengnan Zhao, Zhixuan Chu, Wanyu Lin, Tianhang Zheng

llms agents

arXiv AI

Sep 1

SIR: Self-improving Red-teaming for Compute Use Agents

arXiv:2608.30207v1 Announce Type: cross Abstract: Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal...

By Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho

llms agents multimodal benchmarks safety

arXiv Machine Learning

Sep 10

WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents

arXiv:2605.23220v2 Announce Type: replace Abstract: Despite the growing use of world models as decision-making agents, their adversarial robustness remains underexplored due to the lack of dedicated...

By Zhixiang Guo, Siyuan Liang, Shi Fu, Cheng Guo, Andras Balogh, Mark Jelasity, Dacheng Tao

agents safety

arXiv AI

Aug 11

Query-Only Backdoor Attacks on Self-Evolving Skills via Trajectory Poisoning

arXiv:2608. 08303v1 Announce Type: new Abstract: Agentic skills improve large language model (LLM) agents by encoding reusable procedures for complex tasks.

By Yuyang Luo, Haoran Wang, Kai Shu

llms agents robotics benchmarks

arXiv AI

Sep 2

Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents

arXiv:2609.01487v1 Announce Type: cross Abstract: Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable chan...

By Xiaofang Yang, Ziqi Miao, Dianbo Sui, Jing Shao, Lijun Li

llms agents multimodal benchmarks safety

Related stories