arXiv AI
Guard Models Are Overconfident Where Base Models Are Uncertain
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 2
A unifying Bayesian framework for adversarial robustness
arXiv:2510. 09288v2 Announce Type: replace-cross Abstract: The vulnerability of machine learning models to adversarial attacks remains a critical societal security challenge.
arXiv Machine Learning
Jul 7
Robustness Meets Uncertainty: Evidential Adversarial Training for Robust Selective Classification
arXiv:2607. 03075v1 Announce Type: new Abstract: Safety-critical applications require classifiers that are both robust and reliable.
arXiv Machine Learning
Sep 2
On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification
arXiv:2506.12454v2 Announce Type: replace-cross Abstract: What fundamentally distinguishes an adversarial attack from a misclassification due to limited model expressivity or finite data? In this wor...
arXiv AI
Jun 6
Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
arXiv:2606. 05614v1 Announce Type: new Abstract: Large language models (LLMs) are rigorously aligned to refuse harmful requests, a process that inherently cultivates a latent capacity to evaluate and recognize unsafe content.
OpenAI Blog
Aug 22, 2019
Testing robustness against unforeseen adversaries
We’ve developed a method to assess whether a neural network classifier can reliably defend against adversarial attacks not seen during training. Our method yields a new metric, UAR (Unforeseen Attack Robustness), which evaluates the robustness of a single model against an unanticipated attack, and highlights the need to measure performance across a more diverse range of unforeseen attacks.
arXiv AI
Aug 26
ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models
The paper introduces ADVERSA, an automated red‑teaming framework that evaluates large language model safety over multiple turns by tracking continuous compliance trajectories instead of binary jailbreak outcomes. Using a fine‑tuned 70B attacker model and a structured 5‑point rubric, the authors conduct controlled experiments on three frontier victim models, measuring guardrail degradation and judge reliability through a triple‑judge consensus. Results show a 26.7% jailbreak rate with most breaches occurring early, and the study documents inter‑judge agreement, attacker drift, and attacker refusals as key factors affecting safety assessment.