The paper presents a systematic study of combining defenses against jailbreak attacks on Large Language Models across different pipeline stages. It introduces a standardized evaluation framework that defines attack-success-rate, controls query budgets, and applies explicit fairness rules. Across 19 attacks and 15 defenses, the study finds that no single defense dominates, but carefully selected combinations can provide strong safety with minimal loss of utility, offering practical guidance for layered defense pipelines.
arXiv:2607. 24392v1 Announce Type: cross Abstract: Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility.
AlcaTRAz is a prompt‑level defense that uses rule trees to insert controlled character‑level perturbations into input text, disrupting jailbreak attacks without modifying or retraining the target LLM. It operates solely on the input, making it suitable for black‑box deployments, and was evaluated on 33 open‑weight models and 22 jailbreak types, outperforming three baseline defenses in 73.4 % of model‑attack combinations. While it significantly reduces high‑severity jailbreak success, it does not eliminate it and is intended as one layer of a broader defense strategy.
arXiv:2606. 16751v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks.
arXiv:2508. 10029v3 Announce Type: replace-cross Abstract: Safety-aligned large language models can still be manipulated through white-box interventions that modify their internal representations.
arXiv:2608.21895v1 Announce Type: cross Abstract: Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-serv...
The paper introduces Fair-ASR, a protocol for evaluating black‑box jailbreak attacks using a shared target‑call budget, addressing the bias of prior studies that rely solely on attack success rate. Re‑evaluating 11 attacks under this protocol shows significant shifts in rankings and highlights that many methods are not efficient in both target and attacker calls. The authors then present ReCode, a budget‑efficient attack that combines desensitization rewriting with low‑cost primitives, achieving 85% ASR on GPT‑5 with only 7.19 attacker calls per request under a 20‑target‑call budget.
arXiv:2502. 09755v4 Announce Type: replace-cross Abstract: Safety-aligned LLMs respond to prompts with either compliance or refusal, each corresponding to distinct directions in the model's activation space.
The paper introduces Fair-ASR, a new evaluation protocol for black-box jailbreak attacks that uses shared target-call budgets to provide a fair comparison across methods. Re‑evaluating 11 attacks under this protocol shows that rankings shift significantly with different budgets, and that simple perturbations and templates remain competitive. The authors also present ReCode, a budget‑efficient attack that combines desensitization rewriting with low‑cost primitives, achieving 85% ASR on GPT‑5 with only 7.19 attacker calls per request under a 20‑call budget.
The paper introduces a self‑evolving defense framework for large language models that uses a persistent, cross‑interaction rule memory to adapt to new jailbreak attacks. When an attack succeeds, the system abstracts the failure into a method‑level rule that captures the structural attack wrapper, allowing the rule to generalize across an entire attack family. This memory‑based adaptation operates without parameter updates, works with both open‑weight and black‑box models, and has been shown to reduce attack success rates while preserving benign utility across multiple jailbreak families.
arXiv:2602. 24009v4 Announce Type: replace-cross Abstract: Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols.
arXiv:2606. 03486v1 Announce Type: cross Abstract: Large language models remain vulnerable to jailbreak attacks that hide harmful intent behind seemingly ordinary requests such as role-play, translation, encoding, adversarial suffixes, and multi-turn buildup.
The paper investigates safety risks in model merging, showing that even when all constituent models are individually safety‑aligned, merging can expose a jailbreak vulnerability rooted in the pretrained foundation model. It introduces Basin‑Aware Jailbreak (BAJ), a min–max optimization method that generates adversarial suffixes transferable across merged models sharing the same backbone, without needing the exact merging coefficients or checkpoints. Experiments demonstrate BAJ’s high transfer success rates across diverse backbones and merging settings, and its resilience against existing defenses.