Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade. The judge is rarely checked.
The paper investigates whether automatic safety judges evaluate the content of a model’s reply or merely its style. By keeping the reply content fixed and adding various style wrappers—such as educational disclaimers, fake reasoning blocks, or token refusals—the authors show that many judges flip their verdicts, indicating that style can influence safety judgments. The study evaluates over 600 jailbreak examples across multiple judges, revealing that some judges are highly susceptible to style-based manipulation while others remain robust.
By Yongxi Zhou, Wenbo Ye, Yuanzhe Liu, Zihan Dong, Junwei Yao
The paper investigates whether automatic safety judges evaluate the content of a model’s reply or merely its style. By adding content‑invariant style wrappers—such as educational disclaims or token refusals—to fixed replies, the authors show that many judges flip their verdicts, revealing exploitable blind spots. Across more than 600 jailbreak examples and eight judges, some judges exhibit high flip rates (e.g., GPT‑4o‑mini 19.9%) while others remain largely stable, and human validation confirms that most flips are judge errors rather than content changes.
arXiv:2605. 03226v2 Announce Type: replace-cross Abstract: Safety fine-tuning of language models typically requires a curated adversarial dataset.
By Prakhar Gupta, Garv Shah, Donghua Zhang
arXiv:2606. 02959v1 Announce Type: new Abstract: Published evaluations of prompt-injection and jailbreak detectors for Large Language Models often suffer from two systematic weaknesses: per-dataset threshold tuning and undisclosed operating points.
By Ryle Goehausen, Marcus Sousa
arXiv:2608. 19266v1 Announce Type: cross Abstract: The OWASP Top 10 for LLM Applications ranks the risks that a community of security practitioners judges most important.
By Kyriakos "Rock" Lambros, Steve Wilson