← Back to all news
arXiv AI By Jiongchi Yu, Keke Lian, Junqing Wang, Zhikang Fan, Zhaoxuan Li, Yongheng Huang, Ziming Zhao, Dong Zhang

A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code

Read the original on arXiv AI →

The paper introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs across 51 projects, each run in Docker with functional tests and proof‑of‑concept exploits. It evaluates 13 context‑construction methods and six controls across six LLMs, showing that functionality‑oriented context improves functional pass rates by up to 8.2 pp but only 3.7 pp for security, that omitting safeguards can drop secure pass rates by 8.1 pp, and that coding agents see relevant safeguards in only 27‑36 % of trajectories. These results highlight how repository context shapes the security of LLM‑generated code and guide the design of security‑aware context construction and evaluation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 12

Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code

The paper introduces the Static‑Pass Dynamic‑Fail (SPDF) phenomenon, showing that static analysis can miss vulnerabilities that are exploitable at runtime. Using a three‑stage pipeline—static scanning, LLM‑driven CWE reasoning, and autonomous exploit verification—it evaluated 1,355 Python samples and found that 14.53% of samples that passed static checks were actually exploitable. The study highlights that static‑analysis success and runtime security are distinct assurance layers, especially for AI‑generated and security‑sensitive code.

By Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo
llmsagentsreinforcement-learningsafety
More like this →
arXiv AI
Oct 5

Beyond Predefined Sinks: Security-Aware Dependency Analysis for LLM Agents

The paper introduces AgentSecGraph, a static analysis framework that builds a Security-Aware Agent Dependency Graph (Security-ADG) for each security-sensitive operation in large language model (LLM) agents. It enriches operation identity with agent relevance, source and dependency evidence, trust-boundary context, guard evidence, and external-effect semantics. The authors also present AgentSecBench, a corpus of 67 real-world LLM-agent repositories, and demonstrate that their analyzer identifies thousands of operation candidates, recovers substantial dependency and guard evidence, and distinguishes guarded behaviors from vulnerabilities with high accuracy.

By Hang Cui
llmsagents
More like this →
arXiv Computation and Language
Sep 15

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

arXiv:2407.02395v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but ins...

By Jiexin Wang, Liuwen Cao, Xitong Luo, Yang Cao, Zhenghao Li, Yunyi Xiao, Mengchen Zhao, Adam Jatowt, Yi Cai
llmsagentsbenchmarks
More like this →
arXiv AI
Jun 9

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv:2509. 22097v5 Announce Type: replace-cross Abstract: Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern.

By Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo
llmsagentsbenchmarks
More like this →