Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

The paper examines how the scores of cybersecurity large language model (LLM) benchmarks vary depending on the evaluation pipeline used. By auditing eight benchmarks across ten different LLMs, the authors uncover 15 systematic failure modes and demonstrate that a single pipeline choice can shift a model’s score by over 80 percentage points, significantly altering rankings. They also show that even semantically similar tasks can produce different model rankings due to incompatible evaluation conventions, and that standardizing pipelines can move most models by at least three ranks on at least one benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• benchmarks