arXiv AI

From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation

The paper evaluates seven open‑source small language models (20 M–1.5 B parameters) on two post‑exploitation code‑generation tasks—Directory Enumeration and Backdoor Generation—in both Python and Lua. It finds that offensive code‑generation performance does not scale linearly with model size; instead, task formulation and prompting strategy play a larger role. Using algorithmic formulation and few‑shot prompting, Qwen2.5‑Coder 1.5B achieves near‑perfect success rates on both tasks in Python and high success in Lua.

arXiv AI
Sep 10

SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

SCRIPTIOC-BENCH is a benchmark designed to evaluate how well large language models can statically extract indicators of compromise (IOCs) from script-based malware. It contains 634 manually verified JavaScript, PowerShell, and VBScript samples and covers four IOC types—URLs, domains, IP addresses, and filesystem artifacts—while distinguishing between directly exposed and encoded indicators. Experiments show that even the best models achieve only 65.4 F1, and a false‑positive taxonomy is introduced to analyze error patterns, with two mitigations (deterministic string utilities and task‑specific adaptation) improving precision and shifting errors toward sample‑grounded mismatches.

By Hanna Kim, Jian Cui, Minkyoo Song, Hwanjo Heo, Seungwon Shin, Kimin Lee, Xiaojing Liao
arXiv AI
Sep 12

Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code

The paper introduces the Static‑Pass Dynamic‑Fail (SPDF) phenomenon, showing that static analysis can miss vulnerabilities that are exploitable at runtime. Using a three‑stage pipeline—static scanning, LLM‑driven CWE reasoning, and autonomous exploit verification—it evaluated 1,355 Python samples and found that 14.53% of samples that passed static checks were actually exploitable. The study highlights that static‑analysis success and runtime security are distinct assurance layers, especially for AI‑generated and security‑sensitive code.

By Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo