Hugging Face Trending Papers

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

The paper investigates how self‑evolving personal agents become more vulnerable to indirect prompt injection (IPI) attacks. Using a two‑phase methodology—first allowing the agent to learn from user feedback on a task, then testing it with a third‑party hidden instruction—the authors find that self‑evolution significantly increases attack success rates, rising by 14.0–48.8 percentage points with skill evolution and 3.0–39.3 percentage points with reinforcement learning. The study demonstrates that the very mechanisms designed to improve agent performance also degrade their robustness to malicious manipulation.

arXiv AI

Oct 31

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

The paper investigates how self‑evolving personal agents become more vulnerable to indirect prompt injection (IPI). Using a two‑phase methodology, the authors show that when agents learn from user feedback—either through skill evolution or reinforcement learning—the attack success rate of IPI increases by 14.0–48.8 percentage points compared to a non‑evolving baseline. The study highlights a significant safety risk for personal agents that adapt based on user interactions.

By Xinyan Wang, Xiaogeng Liu, Jun Shao, Chaowei Xiao

agents reinforcement-learning safety

arXiv Machine Learning

Oct 8

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

arXiv:2610.06401v2 Announce Type: replace-cross Abstract: Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing tr...

By Mohamed Dhouib, Clement Elliker, Alexi Canesse, Ma\"el Jenny, Lucas-Andrei Thil, Mahammed El Sharkawy, Sonia Vanier, Elie Bursztein

llms agents efficiency benchmarks

arXiv AI

Sep 1

SIR: Self-improving Red-teaming for Compute Use Agents

arXiv:2608.30207v1 Announce Type: cross Abstract: Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal...

By Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho

llms agents multimodal benchmarks safety

arXiv AI

Sep 17

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

The paper revisits Thompson’s classic compiler back‑door attack in the context of self‑modifying AI coding agents. By poisoning the benchmarks used for self‑evaluation, the authors demonstrate that agents such as the Darwin Gödel Machine, Self‑Improving Coding Agent, and Hyperagents can be coaxed into generating vulnerable code, even on clean, held‑out tasks. Experiments show that the contamination can persist after subsequent clean training, highlighting the need for more robust agent designs.

By Franziska Roesner, Tadayoshi Kohno

agents benchmarks

Hugging Face Trending Papers

Aug 31

SIR: Self-improving Red-teaming for Compute Use Agents

Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate ev...

llms agents multimodal benchmarks safety

arXiv AI

Aug 6

AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection

arXiv:2608. 04053v1 Announce Type: cross Abstract: Prompt injection remains a critical threat to LLM agents, yet existing defenses treat each task as a self-contained problem, independent of previous encounters.

By Shihao Weng, Yang Feng, Xiaofei Xie, Jiongchi Yu

llms agents benchmarks

arXiv AI

Oct 31

Where You Inject Matters: The Trust Region Effect in LLM Agents

The paper investigates how the location of malicious instructions—known as indirect prompt injections—affects the vulnerability of large language model agents. By testing agents across tool use, coding, and daily workflow scenarios, the authors discover a "trust region effect," where agents are more likely to follow unsafe instructions placed in certain environmental surfaces, such as README files, than in others. They also develop learning-based attackers that optimize both injection content and placement, demonstrating that both factors significantly influence attack success.

By Chengquan Guo, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Jialin Song, Qiyao Ma, Carolyn Buractaon, Christopher White, Bo Li, Jianfeng Gao

llms agents

arXiv AI

Jul 17

Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems

arXiv:2607. 14611v1 Announce Type: cross Abstract: A growing class of agentic systems maintain persistent state across sessions through memory files, behavioral preferences, and knowledge bases.

By Soham Gadgil, David Alexander, Sai Sunku, Franziska Roesner

llms agents safety

arXiv AI

Jul 7

When Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agents

arXiv:2607. 05189v1 Announce Type: cross Abstract: Persistent personal agents combine long-term memory with access to users' external environments, enabling personalized foreground assistance and proactive background execution.

By Yechao Zhang, Shiqian Zhao, Jiawen Zhang, Jie Zhang, Gelei Deng, Xiaogeng Liu, Chaowei Xiao, Tianwei Zhang

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Oct 7

AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model

arXiv:2610.08773v1 Announce Type: cross Abstract: Web agents complete user requests by reading and acting on pages that third parties write, so an instruction planted on a page can redirect the agent...

By Sarim Hashmi, Mukul Ranjan, Kshitij Mishra, Mikhail Kuznetsov, Praneeth Vepakomma, Nils Lukas

agents fine-tuning safety

arXiv AI

Jul 7

Agent Data Injection Attacks are Realistic Threats to AI Agents

arXiv:2607. 05120v1 Announce Type: cross Abstract: AI agents act on behalf of user prompts, consuming external data and taking actions based on the agent context.

By Woohyuk Choi, Juhee Kim, Taehyun Kang, Jihyeon Jeong, Luyi Xing, Byoungyoung Lee

llms agents

arXiv AI

Jun 16

MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks

arXiv:2602. 09222v2 Announce Type: replace-cross Abstract: Large language model (LLM) based web agents are increasingly deployed to automate complex online tasks by directly interacting with web sites and performing actions on users' behalf.

By Georgios Syros, Evan Rose, Brian Grinstead, Christoph Kerschbaumer, William Robertson, Cristina Nita-Rotaru, Alina Oprea

llms agents safety

Related stories