arXiv AI August 28, 2026 By Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

The paper evaluates three AI model security scanners—ModelScan, ModelAudit, and Fickling—using a benchmark of 170 Pickle and PyTorch artifacts from 145 families, 135 of which have binary security labels. It distinguishes coverage metrics such as non‑N/A coverage, analysis completion, and definitive security decisions, finding that ModelAudit achieved 100% definitive decisions, Fickling 81.5%, and ModelScan 49.6%. When a definitive judgment was made, ModelScan reached perfect precision, recall, and F1, while Fickling added no unique true positives beyond those found by the other tools.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

benchmarks

arXiv AI

Aug 21

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

arXiv:2608. 19901v1 Announce Type: cross Abstract: Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration.

By Yue Wang, Yi Liu, Gelei Deng, Ying Zhang, Yuekang Li, Zhenyu Chen, Leo Zhang

llms agents benchmarks

arXiv Machine Learning

Aug 11

Activation Probes Surface Code-Security Signals that the Model's Output Misses

arXiv:2608. 09643v1 Announce Type: cross Abstract: AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated.

By Ivan Wiryadi

llms agents

arXiv AI

Jun 18

SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

arXiv:2606. 18356v1 Announce Type: cross Abstract: Tool-using language-model agents introduce security failures that go beyond unsafe text: they can disclose protected objects, write persistent memory, send messages, modify databases, or trigger harmful code and tool effects.

By Yuchuan Tian, Mengyu Zheng, Haocheng Mei, Ye Yuan, Chao Xu, Xinghao Chen, Hanting Chen, Yu Wang

llms agents benchmarks safety

arXiv Machine Learning

Jul 21

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

arXiv:2602. 14161v2 Announce Type: replace Abstract: Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization.

By Max Fomin

llms agents benchmarks safety

arXiv AI

Jun 2

ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree

arXiv:2606. 01494v1 Announce Type: cross Abstract: Agent skills extend AI agents with reusable instructions, tools, scripts, references, and workflows, establishing a security boundary distinct from both model safety and traditional package-malware detection.

By Vincent Koc, Patrick Erichsen, Jacob Tomlinson, Agustin Rivera, Michael Appel, Nir Paz

agents safety

arXiv Machine Learning

Jul 31

Cybersecurity Detection Classification with Reasoning-enabled Language Models

arXiv:2607. 28460v1 Announce Type: new Abstract: A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day.

By Amol Khanna, Manu Nandan, Cristian Viorel Popa, Joan Pujol-Roig, Diana Bolocan, Laura Vasilie, Alexandru Apostu, Chase Helwig, Mihaela Gaman, Michael Brautbar, Edward Raff, Chase Midler, Sven Krasser

llms reinforcement-learning

One email a morning, machine-written

Related stories

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Activation Probes Surface Code-Security Signals that the Model's Output Misses

SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree

Cybersecurity Detection Classification with Reasoning-enabled Language Models