arXiv AI

SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

arXiv:2606. 13757v1 Announce Type: cross Abstract: Large language model (LLM) reviewers are increasingly used in pull-request (PR) workflows, where their approvals help decide which code is merged into a repository.

arXiv AI

Jun 18

Code-Augur: Agentic Vulnerability Detection via Specification Inference

arXiv:2606. 18619v1 Announce Type: cross Abstract: The advent of agentic vulnerability detection is already becoming a watershed moment for software security.

By Zhengxiong Luo, Mehtab Zafar, Dylan Wolff, Abhik Roychoudhury

llms agents benchmarks

arXiv AI

Aug 5

AgenticSCR: An Autonomous Agentic Secure Code Review for Immature Vulnerabilities Detection

arXiv:2601. 19138v2 Announce Type: replace-cross Abstract: Secure code review is critical during pre-integration, where Atlassian developers rely on lightweight analysis tools, while deep security assessment is deferred to later stages, delaying feedback and increasing remediation costs.

By Wachiraphan Charoenwet, Kla Tantithamthavorn, Patanamon Thongtanunam, Hong Yi Lin, Minwoo Jeong, Ming Wu

llms agents

arXiv AI

Jun 3

Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation

arXiv:2606. 03128v1 Announce Type: cross Abstract: Smart contracts face critical security challenges that require thorough auditing in decentralized web services.

By Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra

llms efficiency benchmarks safety

arXiv AI

Aug 10

CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training

arXiv:2608. 06471v1 Announce Type: cross Abstract: Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software.

By Amine Lbath, Manan Suri, Aurelien Delaitre, Vadim Okun, Massih-Reza Amini, Ram D. Sriram, Dinesh Manocha

llms agents fine-tuning

arXiv AI

Jun 4

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

arXiv:2606. 04739v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings.

By Sabrina Kaniewski, Fabian Schmidt, Tobias Heer

llms rag

arXiv Machine Learning

Jul 21

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).

By Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, Lingming Zhang

llms agents benchmarks safety

arXiv AI

Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney

llms agents multimodal benchmarks safety

arXiv AI

Jun 9

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.

By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo

llms agents benchmarks

arXiv AI

Jul 31

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.

By Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li

llms agents nlp benchmarks

arXiv AI

Jul 2

Knowdit: Agentic Smart Contract Vulnerability Detection with Auditing Knowledge Summarization

arXiv:2603. 26270v2 Announce Type: replace-cross Abstract: Smart contracts govern billions of dollars in decentralized finance (DeFi), yet automated vulnerability detection remains challenging because many vulnerabilities are tightly coupled with project-specific business logic.

By Ziqiao Kong, Wanxu Xia, Chong Wang, Yue Xue, Yi Lu, Pan Li, Shaohua Li, Zong Cao, Yang Liu

agents nlp

arXiv AI

Jun 18

SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

arXiv:2606. 18356v1 Announce Type: cross Abstract: Tool-using language-model agents introduce security failures that go beyond unsafe text: they can disclose protected objects, write persistent memory, send messages, modify databases, or trigger harmful code and tool effects.

By Yuchuan Tian, Mengyu Zheng, Haocheng Mei, Ye Yuan, Chao Xu, Xinghao Chen, Hanting Chen, Yu Wang

llms agents benchmarks safety

arXiv Machine Learning

Jun 16

Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

arXiv:2606. 15123v1 Announce Type: cross Abstract: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.

By Yiwei Chen, Lichi Li, Kai Cheung, Vinny Parla, Ganesh Sundaram

llms fine-tuning benchmarks data-engineering

Related stories