arXiv Machine Learning October 19, 2026 By Manohar Vellala, Soumya Banerjee, Sandip Roy, Ross Gore, Sachin Shetty

HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation

Read the original on arXiv Machine Learning →

HardEval is an execution‑verified benchmark that tests language‑model‑assisted security hardening on a live Red Hat Enterprise Linux 8 system. It uses STIG controls that fail on an unhardened baseline, and models generate Bash scripts that are run on a virtual machine and graded by OpenSCAP/OVAL compliance checks. The benchmark reveals significant differences in model performance across security domains and highlights failures where code appears plausible but does not meet the exact compliance condition.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 1

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

arXiv:2606. 30755v1 Announce Type: cross Abstract: Claw-like AI agents (e.

By Peizhi Niu, Wenjie Qu, Shangding Gu, Tianneng Shi, Yuankai Li, Ahmad Tawaha, Hend Alzahrani, Vincent Siu, Boyi Li, Chenguang Wang, Jiaheng Zhang, Basel Alomair, Ming Jin, Muhao Chen, Chi Wang, Costas Spanos, Dawn Song

llms agents benchmarks safety

arXiv AI

Oct 15

Scalable Weak-to-Strong Verifier Hardening for Agent Environments

arXiv:2610.15924v1 Announce Type: cross Abstract: Agent environments are essential for large language model (LLM) development. They use programmatic verifiers to turn complex outcomes into rewards. V...

By Chong Zeng, Ziqian Zhong, Kexun Zhang, Ivan Bercovich, Ivgeni Segal, Yuzhen Mao, Shashwat Saxena, Aditi Raghunathan

llms agents

arXiv AI

Sep 12

Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code

The paper introduces the Static‑Pass Dynamic‑Fail (SPDF) phenomenon, showing that static analysis can miss vulnerabilities that are exploitable at runtime. Using a three‑stage pipeline—static scanning, LLM‑driven CWE reasoning, and autonomous exploit verification—it evaluated 1,355 Python samples and found that 14.53% of samples that passed static checks were actually exploitable. The study highlights that static‑analysis success and runtime security are distinct assurance layers, especially for AI‑generated and security‑sensitive code.

By Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo

llms agents reinforcement-learning safety

arXiv AI

Jun 17

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

arXiv:2510. 01359v2 Announce Type: replace-cross Abstract: Code-capable large language model (LLM) agents are embedded in software engineering workflows where they can read, write, and execute code, raising "jailbreak" stakes beyond text-only settings.

By Shoumik Saha, Jifan Chen, Sam Mayers, Sanjay Krishna Gouda, Zijian Wang, Varun Kumar

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jul 21

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

arXiv:2605. 26548v2 Announce Type: replace-cross Abstract: Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC).

By Hwiwon Lee, Jiawei Liu, Dongjun Kim, Wubing Xia, Ziqi Zhang, Chunqiu Steven Xia, Lingming Zhang

llms agents benchmarks safety

arXiv AI

Jul 22

Data Leakage Prevention in Agentic Applications via Preemptive Hardening

arXiv:2607. 18847v1 Announce Type: cross Abstract: Agentic systems integrate LLM driven planning with interfaces to external tools, making data leakage and tool misuse feasible via instruction/data boundary failures and prompt injection attacks.

By Akansha Shukla, Emily Bellov, Parth Atulbhai Gandhi, Yuval Elovici, Asaf Shabtai

llms agents robotics benchmarks safety

One email a morning, machine-written

Related stories

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

Scalable Weak-to-Strong Verifier Hardening for Agent Environments

Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

Data Leakage Prevention in Agentic Applications via Preemptive Hardening