Simon Willison August 28, 2026

Just a rumour of a bug is enough to find a security exploit these days

The article reports that rumors of bugs in open‑source projects now trigger rapid security exploits, with automated agents probing for vulnerabilities within minutes of a patch being discussed. Anil Madhavapeddy demonstrates how modern coding agents can exploit even minimal hints, and rclone maintainer Nick Craig‑Wood notes a dramatic spike in security disclosures and delayed CVE assignments. The post highlights the need for new processes to protect open‑source communities from swift exploitation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Simon Willison.

arXiv AI

Jun 9

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

arXiv:2606. 08960v1 Announce Type: cross Abstract: Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking.

By Ziqian Zhong, Ivgeni Segal, Ivan Bercovich, Shashwat Saxena, Kexun Zhang, Aditi Raghunathan

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jul 9

AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

arXiv:2604. 11950v2 Announce Type: replace-cross Abstract: While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection.

By Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang

llms rag agents benchmarks safety

arXiv Machine Learning

Jul 17

Agentic Vulnerability Reasoning on COTS Binaries

arXiv:2605. 05000v2 Announce Type: replace-cross Abstract: LLM agents have been increasingly adopted for solving security tasks.

By Hwiwon Lee, Jongseong Kim, Lingming Zhang

llms agents benchmarks

arXiv AI

Jun 15

SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

arXiv:2606. 13757v1 Announce Type: cross Abstract: Large language model (LLM) reviewers are increasingly used in pull-request (PR) workflows, where their approvals help decide which code is merged into a repository.

By Rui Melo, Riccardo Fogliato, Sean Zhou, Pratiksha Thaker, Zhiwei Steven Wu

llms agents benchmarks safety

arXiv AI

Jun 18

Code-Augur: Agentic Vulnerability Detection via Specification Inference

arXiv:2606. 18619v1 Announce Type: cross Abstract: The advent of agentic vulnerability detection is already becoming a watershed moment for software security.

By Zhengxiong Luo, Mehtab Zafar, Dylan Wolff, Abhik Roychoudhury

llms agents benchmarks

arXiv AI

Jun 9

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.

By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo

llms agents benchmarks

One email a morning, machine-written

Related stories

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

Agentic Vulnerability Reasoning on COTS Binaries

SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

Code-Augur: Agentic Vulnerability Detection via Specification Inference

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios