arXiv:2606. 25487v1 Announce Type: cross Abstract: Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade.
By Yang Gao (Veyon Solutions)
Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade. The judge is rarely checked.
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
By Kaihua Ding
arXiv:2609.02899v1 Announce Type: new
Abstract: Benchmark contamination, the leakage of test items into training data, is widely described as a threat to the reliability of large language model (LLM)...
By Xingyao Xiao (Stanford University), Yihong Cheng (City University of Macau)
The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.
By Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato
The paper audits the reproducibility of knowledge‑graph extraction from threat reports by re‑implementing matching rules for only five of twelve systems and re‑scoring ten system outputs under eight protocols. The audit shows that different matching protocols can reverse most pairwise system rankings and that a fixed prediction set can vary from 0.16 to 0.70 F1. The authors also build CTIForge to isolate validation effects, finding that validation changes precision across backbones and increases entity‑type disputes, and they release the full pipeline, protocol suite, and audit records.
By Safayat Bin Hakim, Houbing Herbert Song