Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners
Read the original on arXiv AI →The paper evaluates three AI model security scanners—ModelScan, ModelAudit, and Fickling—using a benchmark of 170 Pickle and PyTorch artifacts from 145 families, 135 of which have binary security labels. It distinguishes coverage metrics such as non‑N/A coverage, analysis completion, and definitive security decisions, finding that ModelAudit achieved 100% definitive decisions, Fickling 81.5%, and ModelScan 49.6%. When a definitive judgment was made, ModelScan reached perfect precision, recall, and F1, while Fickling added no unique true positives beyond those found by the other tools.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.