arXiv AI

Is it my preference? Role Grounding Attack against Agent's Memory via Social Interaction

The paper introduces the Role Grounding Attack (RGA), a method that exploits a vulnerability in LLM-based agents’ memory systems where preferences can be misattributed from an attacker to the victim’s user. By publishing nested conversation posts that quote attacker-user interactions, the attacker causes the victim agent to record the attacker’s preferences as its own, leading to misinformed decisions. The study demonstrates RGA’s effectiveness in tool‑selection manipulation and privacy leakage across multiple memory systems and language models, highlighting the need for stronger role grounding mechanisms.

arXiv Machine Learning

Oct 22

Combining Long-Term Memory and LLMs Comes at a High Cost: From Prompt Injection to Persistent Compromise

The paper investigates a new vulnerability called memory injection, where malicious prompts can persist across sessions in modern LLM‑as‑a‑service platforms. By demonstrating that all NIST attack goals can be achieved on ChatGPT Instant 5.3, Claude Sonnet 4.6, and Gemini 3 Flash, the authors show that these services can spread disinformation, generate harmful content, or cause offline unresponsiveness even in fresh sessions. The study also reveals that many proposed defenses are not yet implemented and provides datasets and artifacts for further research.

By Pedro H. Barcha Correia, Matheus Azevedo de S\'a, Paulo H. Gomes de Senna, Giovanni Mandel Martignago, Victor Takashi Hayashi, Charles Christian Miers, Marcos A. Simplicio Jr

llms rag agents safety

arXiv AI

Jul 9

When Agents Remember Too Much: Memory Poisoning Attacks on Large Language Model Agents

arXiv:2607. 06595v1 Announce Type: cross Abstract: Personal AI agents powered by large language models can reason and act using available tools to access emails, manage calendars, and push code to remote repositories, all with minimal oversight.

By George Torres, Sharad Shrestha, Satyajayant Misra

llms agents benchmarks

arXiv AI

Jun 26

Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents

arXiv:2606. 26627v1 Announce Type: cross Abstract: Large language model agents increasingly query databases, search document collections, call external APIs, remember past interactions, and act on a user's behalf.

By Nada Lahjouji, Ashwin Gerard Colaco

llms rag agents benchmarks safety data-engineering

arXiv AI

Aug 5

Injection-Execution Dissociation: A Mechanistic Evaluation of Persistent Memory Attacks and Defenses in Stateful LLM Agents

arXiv:2605. 08442v5 Announce Type: replace-cross Abstract: We discover that prompt-injection success and tool-execution success are separable safety properties: defenses that block injection do not necessarily block execution, and vice versa.

By Jun Wen Leong

llms rag agents benchmarks safety

arXiv AI

Sep 21

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents

CIPL (Channel Inversion for Privacy Leakage) is a channel-aware framework designed to evaluate black-box privacy leakage in large language model agents. It models the leakage process through stages of sensitive source, selection, assembly, execution, observation, and extraction, assessing how selected sensitive units become attacker-recoverable outputs. Experiments across memory, retrieval, and tool-mediated targets, plus a live-agent case study, reveal that recoverability depends on factors beyond storage labels, such as observation surface, prompt alignment, retrieval depth, and provider behavior, and that a semantic audit can uncover disclosures missed by exact matching.

By Tao Huang, Guosen Wu, Guolong Zheng, Jiayang Meng, Chen Hou, Xu Yang, Xuechao Yang, Feng Xia

llms agents safety

arXiv AI

Sep 1

SIR: Self-improving Red-teaming for Compute Use Agents

arXiv:2608.30207v1 Announce Type: cross Abstract: Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal...

By Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho

llms agents multimodal benchmarks safety

arXiv AI

Aug 12

On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models

arXiv:2608. 10530v1 Announce Type: cross Abstract: Large Language Models (LLMs) have undergone a shift from stateless conversational interfaces to autonomous agents capable of multi-step planning, tool invocation, code execution, and maintaining persistent memory.

By Md Jafrin Hossain, Mohammad Arif Hossain, Nirwan Ansari

llms agents safety

arXiv AI

Oct 28

Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol

The paper "Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol" investigates how natural-language metadata in the Model Context Protocol (MCP) can be exploited to inject malicious post‑hoc logging calls into large language model agents. The authors introduce the Log-To-Leak attack, which adds an attacker‑controlled call after a legitimate tool invocation, targeting user queries, MCP server names, server responses, and agent responses. Experiments across five MCP servers and four LLM agents show high attack success rates (up to 100%) and significant logging accuracy, while existing defenses still leave substantial residual risk.

By Yuepeng Hu, Chongyu Fan, Sirat Samyoun, Jian Du

llms agents multimodal

arXiv AI

Sep 30

Trojan Hippo Bench: A Dynamic Benchmark for Persistent Memory Attacks and Defenses in LLM Agents

arXiv:2605.01970v4 Announce Type: replace-cross Abstract: Memory systems enable otherwise stateless LLM agents to persist user information across sessions, but also introduce a new attack surface. Th...

By Debeshee Das, Julien Piet, Darya Kaviani, Luca Beurer-Kellner, Florian Tram\`er, David Wagner

llms rag agents multimodal benchmarks safety

arXiv Machine Learning

Oct 21

Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory

arXiv:2610.28315v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly use persistent memory to personalize responses, and personal attributes in this memory pose a privacy...

By Masahiro Kaneko, Hiromi Arai, Timothy Baldwin

llms agents safety

arXiv AI

Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney

llms agents multimodal benchmarks safety

arXiv AI

Oct 7

Towards a Unified Misuse Monitoring Benchmark

The paper introduces a unified framework for monitoring misuse of large language model agents in multi‑actor settings, focusing on two attack types: decomposition attacks and prompt injection attacks. It proposes tracking the agent’s externalised actions and determining whether the first detection of harm falls within a defined harm window. Using this formalism, the authors build a benchmark of about 6,200 conversation transcripts and evaluate 17 monitor configurations, finding that action‑framed monitors perform well on both threats while content‑framed monitors struggle with injection attacks, and highlighting limitations of traditional position‑blind metrics.

By Aniruddh Pramod, James Oldfield, Adel Bibi

llms agents benchmarks

Related stories