arXiv:2605. 13138v2 Announce Type: replace-cross Abstract: Automated detection of vulnerability-fixing commits (\vfcs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories.
arXiv:2608. 12246v1 Announce Type: cross Abstract: Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases.
The paper introduces Porting Benchmark, a curated dataset of 1,234 security patch backporting cases that span cross-version, cross-branch, and cross-repository scenarios, along with a common evaluation framework. Five tools—spanning program analysis, LLM prompting, and LLM agents—are evaluated under aligned settings, revealing that performance varies significantly across tools and that complex patches (Type-IV) see a sharp drop in success rate. The study identifies four root-cause categories for failures and demonstrates that reference-based benchmark scores may not fully capture real-world remediation, as executable validation uncovers additional integration issues.
The paper introduces Porting Benchmark, a dataset of 1,234 security patch backporting cases covering cross-version, cross-branch, and cross-repository scenarios, along with a unified evaluation framework. Five tools—spanning program analysis, LLM prompting, and LLM agents—are evaluated under aligned settings, revealing that performance varies significantly across tools and patch complexity, with success rates dropping sharply for structurally complex patches. The study identifies four root-cause categories for failures and demonstrates that reference-based benchmark scores may not fully reflect real-world remediation, as executable validation uncovers additional integration issues.
arXiv:2606. 04739v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings.
The paper introduces DUALLM, a dual-method pipeline that uses a Large Language Model and a fine‑tuned small language model to classify Linux kernel security patches with high precision. By analyzing commit titles, messages, diffs, and code context, DUALLM achieves 87.4% accuracy and an F1‑score of 0.875, outperforming existing methods. It successfully identified 111 recent patches addressing out‑of‑bounds or use‑after‑free vulnerabilities, with 90 confirmed true positives and proof‑of‑concept exploits demonstrating the validity of the classifications.
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions.
arXiv:2606. 17283v1 Announce Type: cross Abstract: Achieving reproducibility, quantity, and diversity in vulnerability datasets has long been viewed as an inherent three-way trade-off, where improving one dimension often comes at the cost of the others.
PatchBench introduces a benchmark to evaluate AI agents on realistic vulnerability patching tasks, addressing two key threats to validity: patch memorization and surface-level fixes that merely suppress crashes. The study finds that 25% of agent patches resemble historical developer patches, and that PoC-only validation inflates success rates by 1.83× on average. PatchBench mitigates these issues by selecting vulnerabilities whose true fixes lie outside the crash stack, migrating historical vulnerabilities into new contexts, and employing rigorous validation for security and semantic correctness.
The paper introduces CodeScan, a black-box, vulnerability-oriented scanning framework designed to detect data poisoning and backdoor attacks in code generation large language models (LLMs). CodeScan operates by analyzing structural similarities across multiple code generations, normalizing them with abstract syntax tree (AST) techniques, and then applying LLM-based vulnerability analysis to identify recurring insecure patterns. Evaluations on 117 models across three architectures and multiple sizes show over 97% detection accuracy with fewer false positives compared to prior methods.
arXiv:2606. 31159v1 Announce Type: cross Abstract: Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure?
arXiv:2608. 15092v1 Announce Type: cross Abstract: In this work, we introduce WeSCE, a benchmark for quantifying security drift in code editing under weak-security constraints, where tasks specify only functional objectives without explicit security requirements.
arXiv:2608. 06471v1 Announce Type: cross Abstract: Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software.