EntailLLM is a framework that verifies vulnerability discovery paths generated by large language model agents using a separate knowledge graph and logic programming. The system checks whether the agent’s traversal of a function call graph is entailed by domain knowledge, improving pooled entailment from 77% to 98% across various binaries and CWE classes. It is deployed on real medical-device binaries, achieving high entailment without per-device tuning and offering logical, explainable verification grounded in formal semantics.
By Kaustuv Mukherji, Jaikrishna Manojkumar Patil, Colton Payne, Paulo Shakarian, Dana Warmsley, Nigel Stepp, Evelyn Kim
The paper examines how Large Language Models (LLMs) have introduced a trustworthiness crisis in cybersecurity vulnerability assessment, producing artifacts known as "AI slop" such as hallucinated vulnerabilities and incorrect patches. It presents a taxonomy of these failures, attributes them to a mismatch between deductive reasoning and probabilistic generation, and introduces the Deductive Coverage Score as a measurable proxy. The authors review mitigation strategies, critique passive approaches, and propose active neuro‑symbolic verification, concluding with new evaluation tools—CVE‑Bench and Slop‑Score—to shift focus from linguistic fluency to mathematical verifiability.
By Junchen Ding, Jialiang Dong, Yichen Zhu, Yi Liu, Gelei Deng, Willy Susilo, Siqi Ma, Yuekang Li
arXiv:2605. 10834v2 Announce Type: replace Abstract: AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets.
By Pedro Conde, Henrique Branquinho, Valerio Mazzone, Bruno Mendes, Andr\'e Baptista, Nuno Moniz
The paper introduces RAS‑Eval, a specification‑based security evaluation framework for large language model agents that uses Linear Temporal Logic (LTL) to define utility and security objectives over tool‑call trajectories and dynamic environment observations. Unlike traditional LLM‑as‑a‑Judge or Agent‑as‑a‑Judge approaches, RAS‑Eval derives deterministic verdicts via a model checker, reducing evaluator uncertainty and enabling reliable assessment over long‑horizon tasks. The authors also present a multi‑agent dataset pipeline and a benchmark of 200 normal and 800 attack tasks covering indirect prompt injection, tool poisoning, skill code injection, and tool code injection.
whyItMatters":"RAS‑Eval addresses key limitations of existing security evaluations—such as evaluator hallucination and missed transient violations—by providing deterministic, environment‑aware verdicts that are crucial for safely deploying LLM agents in unattended, long‑horizon scenarios."
By Yuchuan Fu, Minghe Wang, Xiaohan Yuan, Chenlin Zou, Kaitong Jia, Jinfeng Li, Dongxia Wang
arXiv:2606. 03453v1 Announce Type: cross Abstract: Vulnerability disclosure volumes now far exceed organizational assessment capacity, yet three adjacent research communities (proof-of-concept generation, vulnerability prioritization, and detection rule engineering) operate largely in isolation.
By Farooq Shaikh
arXiv:2604.10800v2 Announce Type: replace-cross
Abstract: Learned classifiers deployed in agentic pipelines face a fundamental reliability problem: predictions are probabilistic inferences, not verif...
By Jugal Gajjar
Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance.
arXiv:2607. 19742v1 Announce Type: cross Abstract: Cyber Threat Intelligence (CTI) reports richly describe real-world attack processes, but their unstructured narratives cannot be directly used for automated attack-path reasoning.
By Wenbo Hou, Ning Hu, Xueping Wang, Jiahao Gu, Wenjian Luo
arXiv:2606. 14517v1 Announce Type: cross Abstract: LLM-based guardrails have emerged as a highly effective defense against prompt injection and jailbreak attacks in autonomous agents.
By Yuguang Zhou, Xunguang Wang, Pingchuan Ma, Zhantong Xue, Zhaoyu Wang, Shuai Wang
The paper proposes a neurosymbolic defense architecture for AI-enhanced Security Operations Centers (AI‑SOCs) that protects against indirect prompt injection via log poisoning. It combines deterministic SIEM decoders as a pre‑filter with NeMo Guardrails for semantic validation, and adds a closed‑loop telemetry system for Human‑in‑the‑Loop visibility. Experimental results mapped to the MITRE ATLAS taxonomy show the approach effectively dismantles promptware kill chains and delivers a resilient, observable defense for next‑generation AI‑SOCs.
By Anna Gazani, Spyridon Kounoupidis, Panagiotis Katsaros, Nikolaos Kekatos, Grigorios Tsoumakas, Georgios Koutidis
arXiv:2606. 18356v1 Announce Type: cross Abstract: Tool-using language-model agents introduce security failures that go beyond unsafe text: they can disclose protected objects, write persistent memory, send messages, modify databases, or trigger harmful code and tool effects.
By Yuchuan Tian, Mengyu Zheng, Haocheng Mei, Ye Yuan, Chao Xu, Xinghao Chen, Hanting Chen, Yu Wang
arXiv:2609. 07344v1 Announce Type: cross Abstract: Large language model (LLM) based agents are increasingly applied to cybersecurity tasks such as vulnerability discovery and automated penetration testing.
By Weizhe Wang, Yitong Zhang, Yao Zhang, Xiaoqiang Di, Zhigang Li, Bin Wu, Guangquan Xu