Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance.
arXiv:2606. 15123v1 Announce Type: cross Abstract: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.
By Yiwei Chen, Lichi Li, Kai Cheung, Vinny Parla, Ganesh Sundaram
arXiv:2607. 19837v1 Announce Type: new Abstract: Traditional pentesting uses reconnaissance at each step to uncover unseen weaknesses, build stronger attacks, and advance the objective; we argue that AI agents require the same treatment.
By Or Zion Eliav, Eyal Lenga, Shir Bernstien, Yisroel Mirsky
arXiv:2609.15523v1 Announce Type: cross
Abstract: Logic attack graphs grounded in scanner output provide explicit and auditable attack path reasoning LLM-based agents lack. Integrating symbolic frame...
By Oliver Stevanovic, Jasmin Wachter
arXiv:2605. 11047v2 Announce Type: replace-cross Abstract: Agentic language-model systems increasingly rely on mutable execution contexts, including files, memory, tools, skills, and auxiliary artifacts, creating security risks beyond explicit user prompts.
By Hongwei Yao, Yiming Liu, Yiling He, Bingrun Yang
arXiv:2606. 14295v1 Announce Type: cross Abstract: Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation.
By Fengyu Liu, Jiarun Dai, Yihe Fan, Wuyuao Mai, Ziao Li, Bofei Chen, Jie Zhang, Zheng Lou, Bocheng Xiang, Qiyi Zhang, Xudong Pan, Geng Hong, Yuan Zhang, Min Yang
The paper proposes using lightweight, calibrated System One decision models—specifically JEV and Laya—to improve autonomous penetration-testing harnesses that rely on large language models (LLMs). It defines four key decision points (finding adjudication, severity recalibration, agent pruning, and confirmation loops) and presents a NeuroSploit case study showing differences in severity distribution, runtime, and grading when using TypeSafe System One. The authors review existing System One specifications, discuss various RL-based training approaches, and introduce Rave, a domain‑adapted model with a proposed training and evaluation framework.
By Joas Antonio dos Santos Barbosa
arXiv:2606. 04460v1 Announce Type: cross Abstract: AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities.
By Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, Francisco De La Riega, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha, Jianhong Tu, Gabriel Han, Chenguang Wang, Jingxuan He, Wenbo Guo, Dawn Song
arXiv:2606. 13079v1 Announce Type: cross Abstract: Nowadays, the autonomous execution of cyberattacks capable of causing substantial real-world harm is widely regarded as one of the critical red lines that frontier AI systems must not cross.
By Jiaqi Luo, Jiarun Dai, Zhile Chen, Jia Xu, Weibing Wang, Yawen Duan, Brian Tse, Geng Hong, Xudong Pan, Yuan Zhang, Min Yang
arXiv:2606. 03453v1 Announce Type: cross Abstract: Vulnerability disclosure volumes now far exceed organizational assessment capacity, yet three adjacent research communities (proof-of-concept generation, vulnerability prioritization, and detection rule engineering) operate largely in isolation.
By Farooq Shaikh
arXiv:2512. 18542v3 Announce Type: replace-cross Abstract: AI coding assistants produce vulnerable code in 45\% of security-relevant scenarios~\cite{veracode2025}, yet no public training dataset teaches both traditional web security and AI/ML-specific defenses in a format suitable for instruction tuning.
By Scott Thornton
The paper introduces CTF-ABACUS, a trace-based auditing framework that reconstructs each autonomous language-model agent’s run in Capture-the-Flag (CTF) challenges into evidence‑grounded solve profiles. By decomposing actions into penetration‑testing phases and techniques, it distinguishes genuine exploitation from shortcut methods such as memorized recall or guessing. Applying the framework to 1,435 CTF attempts by six models on 240 challenges shows that only 62‑87% of recovered flags are trace‑verified, highlighting that many successes rely on shallow trajectories rather than true exploitation.
By Kimberly Milner, Minghao Shao, Nanda Rani, Haoran Xi, Venkata Sai Charan Putrevu, Meet Udeshi, Sandeep K. Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Muhammad Shafique, Ramesh Karri