arXiv Machine Learning

Cross-Corpus Evaluation of Generalizable Vulnerability Detection in IoT Firmware

arXiv:2608. 11492v1 Announce Type: cross Abstract: IoT firmware vulnerability detection remains challenging due to heterogeneous firmware ecosystems, resource-constrained platforms, and limitations in existing benchmarks.

arXiv AI
Sep 21

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

CESBench is a new benchmark for evaluating large language models on cryptographic engineering security for IoT devices, comprising 380 expert‑written items across six sub‑domains such as side‑channel, fault injection, and implementation. The benchmark includes four task types—multiple‑choice, judgment, scenario, and code—each designed to test different competences, with automatic scoring for the first two and LLM‑based judging for the latter two. Evaluation of 11 open‑weight and proprietary LLMs shows strong performance on multiple‑choice and code tasks but weaker results on judgment and scenario tasks, highlighting gaps in justifying security verdicts.

By Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang, Yaoling Ding, Liehuang Zhu
arXiv Machine Learning
Aug 11

Learning to Triage Vulnerability Reports from Program Analysis: An Empirical Study in Node.js

arXiv:2510. 20739v2 Announce Type: replace-cross Abstract: Program analysis tools often produce large volumes of candidate vulnerability reports that require costly manual review, creating a practical challenge: how can security analysts prioritize the reports most likely to be true vulnerabilities?

By Ronghao Ni, Aidan Z. H. Yang, Min-Chien Hsu, Nuno Sabino, Limin Jia, Ruben Martins, Darion Cassel, Kevin Cheang
arXiv AI
Sep 25

Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents

The paper proposes using lightweight, calibrated System One decision models—specifically JEV and Laya—to improve autonomous penetration-testing harnesses that rely on large language models (LLMs). It defines four key decision points (finding adjudication, severity recalibration, agent pruning, and confirmation loops) and presents a NeuroSploit case study showing differences in severity distribution, runtime, and grading when using TypeSafe System One. The authors review existing System One specifications, discuss various RL-based training approaches, and introduce Rave, a domain‑adapted model with a proposed training and evaluation framework.

By Joas Antonio dos Santos Barbosa
arXiv AI
Jun 4

TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering

arXiv:2602. 06911v2 Announce Type: replace-cross Abstract: As increasingly capable open-weight large language models (LLMs) are deployed, improving their tamper resistance against unsafe modifications, whether accidental or intentional, becomes critical to minimize risks.

By Saad Hossain, Tom Tseng, Punya Syon Pandey, Samanvay Vajpayee, Matthew Kowal, Nayeema Nonta, Samuel Simko, Stephen Casper, Zhijing Jin, Kellin Pelrine, Sirisha Rambhatla