arXiv AI
KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 8
SecureCode: A Production-Grade Multi-Turn Dataset for Training Security-Aware Code Generation Models
arXiv:2512. 18542v3 Announce Type: replace-cross Abstract: AI coding assistants produce vulnerable code in 45\% of security-relevant scenarios~\cite{veracode2025}, yet no public training dataset teaches both traditional web security and AI/ML-specific defenses in a format suitable for instruction tuning.
arXiv Computation and Language
Aug 25
CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild
arXiv:2608.23181v1 Announce Type: cross Abstract: As large language models (LLMs) continue to advance in coding capabilities, their potential in cybersecurity has drawn increasing research attention,...
arXiv AI
Jul 7
Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation
arXiv:2603. 17673v2 Announce Type: replace-cross Abstract: LLM agents are becoming increasingly important in the security domain, but leading systems are often closed-source, cloud-based, hard to reproduce or use with sensitive code.
arXiv Machine Learning
Jul 21
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).
arXiv AI
Jul 15
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
arXiv:2605. 10834v2 Announce Type: replace Abstract: AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets.
arXiv AI
Jul 31
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.