arXiv Machine Learning

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).

arXiv Machine Learning
Sep 10

VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

arXiv:2609.08040v1 Announce Type: cross Abstract: The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing d...

By Jiahao Shi, Edward Tsien, Yifeng Di, Hongjiao Zhang, Yuan Tang, Ronit Dey, Ilona Shishov, Gal Netanel, Zvi Grinberg, Vladimir Belousov, Bat-Zion Rotman, Ilan Pinto, Tianyi Zhang
arXiv Machine Learning
Aug 27

FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

FuzzingBrain‑Bench V1 is a new benchmark that tests large language models (LLMs) on their ability to discover software bugs in open‑source projects. Unlike prior benchmarks that focus on a single target vulnerability, this benchmark gives models a Docker‑based harness and asks them to generate inputs that trigger as many distinct crashes as possible. The first version contains 77 challenges from 43 projects (36 C, 32 C++, 9 Java/JVM) and evaluates Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8, with Claude Opus 4.8 achieving the highest score by triggering crashes in 60 of 77 challenges.

By Ze Sheng, Aleksandar Kezic, Zhicheng Chen, Jeff Huang
arXiv Machine Learning
5d ago

What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs

The paper introduces DUALLM, a dual-method pipeline that uses a Large Language Model and a fine‑tuned small language model to classify Linux kernel security patches with high precision. By analyzing commit titles, messages, diffs, and code context, DUALLM achieves 87.4% accuracy and an F1‑score of 0.875, outperforming existing methods. It successfully identified 111 recent patches addressing out‑of‑bounds or use‑after‑free vulnerabilities, with 90 confirmed true positives and proof‑of‑concept exploits demonstrating the validity of the classifications.

By Xingyu Li (UC Riverside), Juefei Pu (UC Riverside), Yifan Wu (UC Riverside), Xiaochen Zou (UC Riverside), Shitong Zhu (UC Riverside), Qiushi Wu (UC Riverside), Zheng Zhang (UC Riverside), Joshua Hsu (UC Riverside), Yue Dong (UC Riverside), Zhiyun Qian (UC Riverside), Kangjie Lu (UC Riverside), Trent Jaeger (UC Riverside), Michael De Lucia (UC Riverside), Srikanth V. Krishnamurthy (UC Riverside)
arXiv AI
Jun 9

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.

By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo
arXiv Machine Learning
Jun 18

OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing

arXiv:2606. 19149v1 Announce Type: cross Abstract: Automated vulnerability discovery in large codebases remains challenging: traditional static analysis produces high false-positive rates, while dynamic approaches such as fuzzing require substantial infrastructure and often target narrow classes of bugs.

By Nahum Korda, Gadi Evron
arXiv AI
Sep 11

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

CS-Guard is a new benchmark that systematically evaluates guardrails for code generation security, covering 1,000 malware-generation prompts, 7 jailbreak attacks, and a novel fictional scenario attack (FSA) for text-to-code generation, as well as 331 code prompts for code-to-code generation. The study empirically tests nine guardrails across seven large language models, finding that many guardrails fail to prevent malicious code generation, with attack success rates reaching about 50% for text-to-code and up to nearly 100% for code-to-code and FSA scenarios. CS-Guard introduces a modular three-layer guardrail taxonomy and releases its benchmark and data to support future research.

By Jinyang Li, Mingyu Guo, Hung X. Nguyen