arXiv AI

Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

arXiv:2509. 14335v2 Announce Type: replace-cross Abstract: Automated malware classifiers achieve strong detection performance, but auditing requires more than flagging a sample: analysts must explain malicious behaviors and justify them with code evidence.

arXiv Machine Learning

Sep 18

ALIBI: Adversarial Legitimacy Injection in Binary Input against LLM Malware Analyzers

The paper introduces ALIBI, a semantic cover story attack that injects a small, non-executed read‑only section into compiled binaries to mislead large language model (LLM) malware analyzers. By embedding a coherent but false security narrative, ALIBI can cause LLMs such as Gemini 2.5 Pro, GPT‑5.5 Pro, and Claude Opus 4.7 to downgrade or flip the verdicts of malicious samples. The attack also transfers to ELF binaries, and even a verification‑guided defense prompt only partially mitigates the effect, leaving a significant portion of malicious samples classified as benign.

By Hyeongjun Choi, Wonyoung Jung, Haehoon Seo, Sungyup Nam

llms safety

arXiv AI

Jul 23

Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

arXiv:2607. 20216v1 Announce Type: cross Abstract: Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours.

By Adel ElZemity, Shujun Li, Budi Arief

llms agents benchmarks safety

arXiv AI

3d ago

SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs

arXiv:2609.36879v1 Announce Type: cross Abstract: As LLM-based agents perform increasingly complex tasks, Agent Skills have emerged as a flexible mechanism for extending their capabilities. An Agent...

By Haoran Ou, Gelei Deng, Xuanye Zhang, Wenbo Guo, Tianwei Zhang, Kwok-Yan Lam

llms agents efficiency

arXiv AI

Sep 25

Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

The paper introduces CodeScan, a black-box, vulnerability-oriented scanning framework designed to detect data poisoning and backdoor attacks in code generation large language models (LLMs). CodeScan operates by analyzing structural similarities across multiple code generations, normalizing them with abstract syntax tree (AST) techniques, and then applying LLM-based vulnerability analysis to identify recurring insecure patterns. Evaluations on 117 models across three architectures and multiple sizes show over 97% detection accuracy with fewer false positives compared to prior methods.

By Shenao Yan, Shan Jin, Shimaa Ahmed, Sunpreet Singh Arora, Yiwei Cai, Yizhen Wang, Yuan Hong

llms

arXiv AI

Jun 19

Multi-View Decompilation for LLM-Based Malware Classification

arXiv:2606. 20436v1 Announce Type: cross Abstract: Malware analysts often inspect compiled binaries through decompiled pseudo-C, when source code is unavailable.

By Bercan Turkmen, Vyas Raina

llms benchmarks

arXiv AI

Jun 2

Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

arXiv:2606. 00925v1 Announce Type: cross Abstract: Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime.

By Ismail Hossain, Sai Puppala, Zhuoran Lu, Sajedul Talukder, Nan Jiang

agents benchmarks

arXiv AI

Aug 11

SkillsMetric: Mapping the Detection Boundary of Static Analysis for Malicious Agent Skills

arXiv:2608. 08468v1 Announce Type: cross Abstract: Agent Skills---structured packages of instructions and scripts that augment LLM-based agents---are rapidly proliferating, yet their security properties remain under-explored.

By Xinze Chen, Chi Zhang, Ping Ji, Yimin Liu

llms agents robotics safety

arXiv AI

Aug 13

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

arXiv:2608. 11469v1 Announce Type: cross Abstract: AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries.

By Jeremy Spence, Nicholas Assaderaghi, Jinhao Zhu, Nikil Ravi, Raluca Ada Popa, Guannan Wei, Yangruibo Ding, Zhuo Zhang

llms agents benchmarks

arXiv AI

Sep 15

ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents

arXiv:2609.14987v1 Announce Type: cross Abstract: Large language model (LLM) agents interact with external environments through tool invocation, but tool outputs can also expose them to indirect prom...

By Bingzheng Wang, Xiaoyan Gu, Wentao Wang, Xingyou Yang, Hongcheng Li, Rong Yin

llms agents benchmarks

arXiv AI

Jun 18

Code-Augur: Agentic Vulnerability Detection via Specification Inference

arXiv:2606. 18619v1 Announce Type: cross Abstract: The advent of agentic vulnerability detection is already becoming a watershed moment for software security.

By Zhengxiong Luo, Mehtab Zafar, Dylan Wolff, Abhik Roychoudhury

llms agents benchmarks

arXiv AI

Sep 10

SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

SCRIPTIOC-BENCH is a benchmark designed to evaluate how well large language models can statically extract indicators of compromise (IOCs) from script-based malware. It contains 634 manually verified JavaScript, PowerShell, and VBScript samples and covers four IOC types—URLs, domains, IP addresses, and filesystem artifacts—while distinguishing between directly exposed and encoded indicators. Experiments show that even the best models achieve only 65.4 F1, and a false‑positive taxonomy is introduced to analyze error patterns, with two mitigations (deterministic string utilities and task‑specific adaptation) improving precision and shifting errors toward sample‑grounded mismatches.

By Hanna Kim, Jian Cui, Minkyoo Song, Hwanjo Heo, Seungwon Shin, Kimin Lee, Xiaojing Liao

llms benchmarks

arXiv Machine Learning

Jul 30

ToxScreen: Detecting Whether an LLM Has Been Poisoned

arXiv:2607. 26849v1 Announce Type: cross Abstract: As large language models (LLMs) are deployed in high-stakes domains, adversaries may poison training data to implant backdoors: hidden triggers that covertly manipulate model behavior at inference time.

By Anthony Hughes, Nicole Xing, Collin Francel, Andy Kim, Andrew Draganov

llms benchmarks

Related stories