← Back to all news
arXiv AI

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

The paper revisits Thompson’s classic compiler back‑door attack in the context of self‑modifying AI coding agents. By poisoning the benchmarks used for self‑evaluation, the authors demonstrate that agents such as the Darwin Gödel Machine, Self‑Improving Coding Agent, and Hyperagents can be coaxed into generating vulnerable code, even on clean, held‑out tasks. Experiments show that the contamination can persist after subsequent clean training, highlighting the need for more robust agent designs.

Related stories

arXiv AI
Oct 27

NameTrap: Steering AI Coding Agents Toward Security Vulnerabilities via Variable Names

The paper introduces NameTrap, a source‑code poisoning attack that renames identifiers in third‑party Python files to steer AI coding agents toward specific Common Weakness Enumeration (CWE) vulnerabilities. By iteratively aligning variable names with CWE‑specific conventions, the attack achieves success rates ranging from 22.5% to 75.0% across six agents and four vulnerability categories, outperforming random and neutral renaming strategies. The poisoned files contain no explicit instructions or vulnerable statements, and existing AI‑based and static security analyzers fail to detect the induced weaknesses.

By Chieh-Yuan Chang, Guan-Wei Chen, Chia-Yi Hsu, Chih Yang, Chia-Mu Yu, Chun-Ying Huang
llmsagentsrobotics
More like this →
arXiv AI
Oct 31

Where You Inject Matters: The Trust Region Effect in LLM Agents

The paper investigates how the location of malicious instructions—known as indirect prompt injections—affects the vulnerability of large language model agents. By testing agents across tool use, coding, and daily workflow scenarios, the authors discover a "trust region effect," where agents are more likely to follow unsafe instructions placed in certain environmental surfaces, such as README files, than in others. They also develop learning-based attackers that optimize both injection content and placement, demonstrating that both factors significantly influence attack success.

By Chengquan Guo, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Jialin Song, Qiyao Ma, Carolyn Buractaon, Christopher White, Bo Li, Jianfeng Gao
llmsagents
More like this →
arXiv AI
Sep 30

Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery

The paper introduces RedHerring, a defense mechanism that inserts safe decoy vulnerabilities into code repositories to divert autonomous LLM agents’ verification efforts away from real security flaws. By embedding CVE-derived vulnerability chains with false bridges and providing a private certificate for quick verification, RedHerring forces agents to spend a significant portion of their limited resources on decoys. Experiments on 33 OSS‑Fuzz projects show a 38.7‑60.4% reduction in discovered real vulnerabilities, even when agents are aware of decoys.

By Kaikai Zhang, Zihan Zhang, Yuchong Xie, Zesen Liu, Shuangjie Yao, Zhixiang Zhang, Dongdong She
llmsagents
More like this →
arXiv AI
Oct 12

Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

The paper examines how fine‑tuned defenses against prompt injection in large language models can be audited before deployment. By leveraging intermediate checkpoints from the fine‑tuning process, the authors create strong optimization‑based attacks (e.g., GCG and BEAST) that dramatically increase attack success rates—from near zero to up to 96%—against state‑of‑the‑art defenses. This demonstrates that current empirical evaluations often overestimate robustness and that pre‑deployment stress‑testing can uncover real vulnerabilities.

By Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus, Yves-Alexandre de Montjoye
llmsfine-tuningbenchmarks
More like this →