arXiv Machine Learning

Cyber-Zero: Training Cybersecurity Agents without Runtime

Cyber-Zero introduces the first runtime‑free framework for training cybersecurity agents by synthesizing high‑quality agent trajectories using publicly available CTF writeups and persona‑driven LLM simulation. The approach generates realistic, long‑horizon interaction sequences without requiring actual execution environments, enabling LLM‑based agents to achieve up to 11.4% absolute performance gains on three major CTF benchmarks. The best model, Cyber‑Zero‑32B, matches proprietary systems such as DeepSeek‑V3‑0324 and Claude‑3.5‑Sonnet while being more cost‑effective, demonstrating that runtime‑free trajectory synthesis can democratize state‑of‑the‑art cybersecurity agent development.

arXiv AI
Oct 2

KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

KaliBench is a fine‑grained benchmark that evaluates large language models’ ability to translate natural‑language queries into executable command‑line instructions for Kali Linux. It contains 8,504 query–command pairs covering 1,642 tools across 23 capability dimensions and 5 security phases, and uses a deterministic, alias‑aware evaluation pipeline with multi‑stage verification (LLM validation, sandboxed execution, and human refinement). The benchmark also supports runtime‑free verifiable rewards, enabling training improvements that bring smaller models close to the performance of a 685B MoE model.

By Pengfei Li, Naufal Suryanto, Sicheng Zhang, Muzammal Naseer
arXiv AI
Jun 12

The Emergence of Autonomous Penetration Capabilities in Large Language Model-Powered AI Systems

arXiv:2606. 13079v1 Announce Type: cross Abstract: Nowadays, the autonomous execution of cyberattacks capable of causing substantial real-world harm is widely regarded as one of the critical red lines that frontier AI systems must not cross.

By Jiaqi Luo, Jiarun Dai, Zhile Chen, Jia Xu, Weibing Wang, Yawen Duan, Brian Tse, Geng Hong, Xudong Pan, Yuan Zhang, Min Yang
arXiv AI
Jul 3

Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens

arXiv:2507. 02964v2 Announce Type: replace-cross Abstract: The increasing scale of AI workloads demands High-Performance Computing (HPC) infrastructure and training methodologies that are both scalable and sustainable.

By Salahuddin Salahuddin, Ahmed Hussain, Jussi L\"opp\"onen, Toni Jutila
arXiv Computation and Language
Nov 4

AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Web Penetration Agents

AgentSnare is a trajectory‑adaptive deception system that expands a decoy environment while autonomous web‑penetration agents explore. It uses an LLM‑driven decoy constructor and a virtual kernel to generate and materialize decoy artifacts that persist across interactions, enabling long‑horizon exploration. In experiments on 15 CVE‑Bench web applications and four attacker models, AgentSnare delays 49.9% of tool calls, diverts 57.2% of post‑entry actions into decoys, and defuses 89.5% of completion reports, preventing any real target exploitation at pass@3 across 60 attacker‑CVE pairs.

By Ruoyu Wang, Heng Zhao, Renjie Wu, Tianhang Zheng, Mengnan Zhao, Wanyu Lin, Kui Ren
arXiv AI
Aug 6

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

arXiv:2608. 04317v1 Announce Type: cross Abstract: Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied.

By Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi, Sanggeon Yun, Hyunwoo Oh, SungHeon Jeong, Nathaniel D. Bastian, Mahdi Imani, Mohsen Imani
arXiv AI
Jul 16

AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.

By Zichen Ding, Jiaye Ge, Shufan Jiang, Kai Chen, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu