arXiv AI

Evaluating LLM Generated Detection Rules in Cybersecurity

arXiv:2509. 16749v1 Announce Type: cross Abstract: LLMs are increasingly pervasive in the security environment, with limited measures of their effectiveness, which limits trust and usefulness to security practitioners.

arXiv AI

Jun 24

FALCON: Transforming Cyber Threat Intelligence into Deployable IDS Rules with Self-Reflection

arXiv:2508. 18684v2 Announce Type: replace-cross Abstract: Signature-based Intrusion Detection Systems (IDS) detect malicious activity by matching network or host events against predefined rules.

By Shaswata Mitra, Subash Neupane, Martin Duclos, Sudip Mittal, Aritran Piplai, Md Rayhanur Rahman, Edward Zieglar, Shahram Rahimi

llms rag agents safety

arXiv AI

Jun 6

GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

arXiv:2606. 05844v1 Announce Type: cross Abstract: Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, signature-driven rules limit adaptability to emerging and zero-day threats.

By Hassan Jalil Hadi, Rehana Yasmin, Ali Shoker

llms agents benchmarks

arXiv AI

Jul 9

Large Language Models (LLMs) and Generative AI in Cybersecurity and Privacy: A Survey of Dual-Use Risks, AI-Generated Malware, Explainability, and Defensive Strategies

arXiv:2607. 06963v1 Announce Type: cross Abstract: Large Language Models (LLMs) and generative AI (GenAI) systems, such as ChatGPT, Claude, Gemini, LLaMA, Copilot, Stable Diffusion by OpenAI, Anthropic, Google, Meta, Microsoft, Stability AI, respectively, are revolutionizing cybersecurity, enabling both automated defense and sophisticated attacks.

By Kiarash Ahi, Saeed Valizadeh

llms diffusion benchmarks safety

arXiv AI

Jul 1

AI-Generated PowerShell Malware: An Experimental Framework and Dataset

arXiv:2606. 30819v1 Announce Type: cross Abstract: Generative AI has emerged as a significant cybersecurity threat, with several recent attack campaigns leveraging LLMs to generate code for malicious purposes via scripting languages such as PowerShell.

By Luciano Pianese, Vittorio Orbinato, Pietro Liguori, Roberto Natella

llms

arXiv AI

Jul 7

SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses

arXiv:2510. 15476v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly used as interfaces to information, code, and real-world services, making prompt-level security failures a practical concern.

By Hanbin Hong, Shuang Wu, Shuya Feng, Nima Naderloui, Shenao Yan, Jingyu Zhang, Ali Arastehfard, Heqing Huang, Yuan Hong

llms benchmarks safety

arXiv AI

Jun 4

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

arXiv:2606. 04460v1 Announce Type: cross Abstract: AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities.

By Tianneng Shi, Robin Rheem, Dongwei Jiang, Mona Wang, Francisco De La Riega, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha, Jianhong Tu, Gabriel Han, Chenguang Wang, Jingxuan He, Wenbo Guo, Dawn Song

agents benchmarks

Hugging Face Trending Papers

Jul 8

Large Language Models (LLMs) and Generative AI in Cybersecurity and Privacy: A Survey of Dual-Use Risks, AI-Generated Malware, Explainability, and Defensive Strategies

Large Language Models (LLMs) and generative AI (GenAI) systems, such as ChatGPT, Claude, Gemini, LLaMA, Copilot, Stable Diffusion by OpenAI, Anthropic, Google, Meta, Microsoft, Stability AI, respectively, are revolutionizing cybersecurity, enabling both automated defense and sophisticated attacks. These technologies power real-time threat detection, phishing defense, secure code generation, and vulnerability exploitation at unprecedented scales.

llms diffusion benchmarks safety

arXiv Machine Learning

Jun 16

Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

arXiv:2606. 15123v1 Announce Type: cross Abstract: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.

By Yiwei Chen, Lichi Li, Kai Cheung, Vinny Parla, Ganesh Sundaram

llms fine-tuning benchmarks data-engineering

arXiv AI

Jun 6

From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability

arXiv:2606. 05252v1 Announce Type: cross Abstract: Security teams routinely simulate attacks against their own systems to check whether their monitoring would catch a real intruder.

By Alexandre Cristov\~ao Maiorano

llms

arXiv AI

Jul 31

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.

By Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li

llms agents nlp benchmarks

arXiv AI

Jul 28

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

arXiv:2607. 23088v1 Announce Type: cross Abstract: Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored.

By Lixun Ma, Ruolong Ma, Bei Wang, Feng Wei, Zhenguang Liu, Lorenzo Cavallaro, Wentao Chen

llms benchmarks

arXiv AI

Jun 11

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

arXiv:2606. 11672v1 Announce Type: cross Abstract: This paper explores the value of agentic AI tools for cybersecurity purposes.

By Derek Yohn, Luke Flancher, Mirajul Islam, Khaled Slhoub

llms agents reinforcement-learning

Related stories