The study compares deterministic static application security testing (SAST) with AI coding agents and AI security harnesses using the VulnBench 2.0 benchmark of 108 vulnerabilities across multiple languages. Nine productized configurations were evaluated, with Snyk Code achieving 82.4% macro recall and Claude Code at 79.3%; together they covered 101 of 108 vulnerabilities, exceeding any single configuration. The results show that AI engines’ effectiveness depends heavily on their harness policies and that AI tools can uncover real vulnerabilities absent from the reference set.
arXiv:2608. 11469v1 Announce Type: cross Abstract: AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries.
By Jeremy Spence, Nicholas Assaderaghi, Jinhao Zhu, Nikil Ravi, Raluca Ada Popa, Guannan Wei, Yangruibo Ding, Zhuo Zhang
arXiv:2606. 15762v1 Announce Type: cross Abstract: We ran 300 repeated vulnerability-finding scans to measure how repeatable agentic large language model (LLM) security review is on the same JavaScript code, prompt, and benchmark harness.
By Liran Tal, Johannes Kloos, Arsenii Rudich, Stephen Thoemmes, Manoj Nair
arXiv:2607. 20759v1 Announce Type: cross Abstract: AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools.
By Ankur Singh, Jinqiu Yang, Tse-Hsun Chen
arXiv:2605. 09504v2 Announce Type: replace-cross Abstract: We present swarm-attack, an open-source adversarial testing framework in which multiple lightweight LLM agents coordinate through shared memory, parallel exploration, and evolutionary optimization.
By Michael A. Riegler, Inga Str\"umke
arXiv:2609.08040v1 Announce Type: cross
Abstract: The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing d...
By Jiahao Shi, Edward Tsien, Yifeng Di, Hongjiao Zhang, Yuan Tang, Ronit Dey, Ilona Shishov, Gal Netanel, Zvi Grinberg, Vladimir Belousov, Bat-Zion Rotman, Ilan Pinto, Tianyi Zhang
arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).
By Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, Lingming Zhang
arXiv:2607. 27030v1 Announce Type: cross Abstract: LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS.
By Petr Simecek, Elnaz Babayeva, Jiri Balhar, Michal Bida, Michal Buran, Vaclav Cadek, Luigino Camastra, Tomas Dulka, Michal Janocko, Tomas Klohna, Pavel Kohout, Ondrej Kokes, Adam Krivka, Jakub Kubik, Patrik Mada, Igor Morgenstern, Marek Pavelka, Joshua Rogers, Petr Stastny, Jan Tattermusch, Dmitrijs Trizna, Martin Votruba, Guido Vranken, Jakub Zikl, Evelina Gabasova, Stanislav Fort
arXiv:2610.17020v1 Announce Type: cross
Abstract: Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An...
By Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain, Guram Gogia, Soumyadeep Bakshi, Nazneen Rajani
arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.
By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo
The paper introduces the Static‑Pass Dynamic‑Fail (SPDF) phenomenon, showing that static analysis can miss vulnerabilities that are exploitable at runtime. Using a three‑stage pipeline—static scanning, LLM‑driven CWE reasoning, and autonomous exploit verification—it evaluated 1,355 Python samples and found that 14.53% of samples that passed static checks were actually exploitable. The study highlights that static‑analysis success and runtime security are distinct assurance layers, especially for AI‑generated and security‑sensitive code.
By Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo
arXiv:2608. 16187v1 Announce Type: cross Abstract: AI-assisted development tools generate vulnerable code at significant rates, yet few automated mechanisms exist to detect, enrich, fix, and verify security issues at development velocity, particularly ones that ground remediation in real-world threat context.
By Mikhail Surikov