arXiv Machine Learning July 30, 2026 By Petr Simecek, Elnaz Babayeva, Jiri Balhar, Michal Bida, Michal Buran, Vaclav Cadek, Luigino Camastra, Tomas Dulka, Michal Janocko, Tomas Klohna, Pavel Kohout, Ondrej Kokes, Adam Krivka, Jakub Kubik, Patrik Mada, Igor Morgenstern, Marek Pavelka, Joshua Rogers, Petr Stastny, Jan Tattermusch, Dmitrijs Trizna, Martin Votruba, Guido Vranken, Jakub Zikl, Evelina Gabasova, Stanislav Fort

HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

Read the original on arXiv Machine Learning →

arXiv:2607. 27030v1 Announce Type: cross Abstract: LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 7

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

arXiv:2607. 02825v1 Announce Type: cross Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection.

By Norbert Sandor Szolnoki, Gabor Antal

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 21

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).

By Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, Lingming Zhang

llms agents benchmarks safety

arXiv AI

Jun 11

Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

arXiv:2605. 23243v2 Announce Type: replace-cross Abstract: We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark: white-box function-level vulnerability detection (VulnLLM-R, across C/Java/Python) and black-box web application security testing (five production-style applications with 118 ground-truth vulnerabilities across 20+ CWE families, which we will open-source).

By Vivek Dahiya, Sunny Nehra, Vipul Dholariya, Bhavik Shangari, Chandra Khatri

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 4

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

arXiv:2606. 04739v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings.

By Sabrina Kaniewski, Fabian Schmidt, Tobias Heer

llms rag

arXiv AI

4d ago

Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

arXiv:2609.37669v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) is increasingly used to enhance Large Language Model (LLM)-based software vulnerability detection by grounding p...

By Sabrina Kaniewski, Tim Kr\"amer, Julius B\"achle, Markus Enzweiler, Michael Menth, Tobias Heer

llms rag benchmarks safety

arXiv AI

Jun 19

Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

arXiv:2606. 20502v1 Announce Type: cross Abstract: Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved.

By Arastoo Zibaeirad, Marco Vieira

llms fine-tuning benchmarks

One email a morning, machine-written

Related stories

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection

Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software