arXiv Machine Learning
Jun 5
Steering Vectors are an Adversarial Attack Surface
arXiv:2606. 05958v1 Announce Type: new Abstract: Activation steering has become a popular way to control Large Language Model (LLM) behavior without fine-tuning.
Bypassing Prompt Guards in Production with Controlled-Release Prompting
arXiv:2510. 01529v3 Announce Type: replace Abstract: Ball et al.
Related stories
arXiv Machine Learning
Jun 30
FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks
arXiv:2606. 28962v1 Announce Type: cross Abstract: Model quantization is essential for the efficient deployment of Large Language Models (LLMs), but introduces a critical vulnerability: Quantization-Conditioned Backdoor (QCB) attacks.
arXiv Computation and Language
6d ago
From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
The paper investigates prompt injection attacks on 14 open‑source and 3 closed‑source large language models (LLMs), introducing a new metric called Attack Success Probability (ASP) that accounts for uncertainty in model responses. It demonstrates that a simple hypnotism attack can trigger objectionable behavior in models such as StableLM2, Mistral, Openchat, and Vicuna, achieving roughly 90% ASP. The study highlights that moderately well‑known LLMs are particularly vulnerable, underscoring the importance of public awareness and effective mitigation strategies.
arXiv AI
Sep 11
Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuning
The paper reports that large language models can acquire cipher-based covert communication skills without fine‑tuning, using prompting or in‑context learning instead. This enables new jailbreak attacks that bypass alignment safeguards by encrypting harmful requests, making them appear as nonsensical text to harmfulness classifiers. The authors demonstrate successful attacks against frontier models from Anthropic, Google, and OpenAI.
arXiv AI
Aug 11
Stealing Reasoning Traces from Proprietary LLM APIs
arXiv:2608. 09867v1 Announce Type: cross Abstract: Leading large language model providers now conceal their models' step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage.
arXiv Machine Learning
Aug 31
Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification
The paper introduces a privacy‑preserving zk‑SNARK audit framework that uses adversarial‑style probes to detect logit drift between an approved large language model and a modified deployment. It offers three probe families—token‑based (black‑box), embedding‑based (gray‑box), and stress probes (partial white‑box)—allowing users to balance sensitivity, access, and cost. Experiments across LLM architectures and GPU platforms show token‑based probes achieve the highest mean sensitivity while remaining practical in a black‑box setting, with Groth16 proving times scaling modestly from 1.02 to 1.78 seconds and constant proof size.
arXiv Machine Learning
Aug 27
Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation
The paper surveys attacks and defenses in Retrieval-Augmented Generation (RAG), a technique that improves large language models by grounding outputs in external knowledge. It identifies new robustness and security risks such as corpus poisoning, backdoor attacks, privacy leakage, and fairness violations, and notes that existing surveys inadequately cover attacker objectives, threat models, and stage-specific defenses. The survey offers a unified, pipeline-aware overview, formalizing threat models across the corpus, retriever, and generator, categorizing attacks by accuracy, privacy, and fairness, and reviewing defenses for retrieval, rerank, generation, and traceback stages, while also summarizing robustness benchmarks and explainability methods.
arXiv AI
Sep 23
Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection
arXiv:2609.24801v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in production systems, raising concerns about their exposure to adversarial manipulation throu...
arXiv AI
2d ago
UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models
UniGuardian is a training‑free detector for large language models that jointly identifies prompt injection, backdoor, and adversarial attacks—collectively called Prompt Trigger Attacks (PTA). It measures how structured prompt perturbations shift the model’s output distribution and uses a single‑forward strategy to detect attacks while generating text in a shared batched forward pass. Experiments show that UniGuardian accurately and efficiently identifies trigger‑activated prompts in LLMs.
arXiv AI
Sep 18
MAS-Shield: A Defense Framework for Secure and Efficient LLM MAS
MAS-Shield is a defense framework for Large Language Model–based Multi-Agent Systems that uses a coarse‑to‑fine filtering pipeline. It first selects critical agents, then applies lightweight auditing to most cases, and finally escalates only suspicious signals to a heavyweight committee. Experiments show a 92.5% recovery rate against adversarial attacks and a latency reduction of over 70% compared to existing methods.
arXiv AI
Jul 2
Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
arXiv:2607. 00481v1 Announce Type: cross Abstract: Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs).
arXiv AI
Jul 23
Matching Ranks Over Probability Yields Truly Deep Safety Alignment
arXiv:2512. 05518v2 Announce Type: replace-cross Abstract: Open-source Large Language Models (LLMs) play a critical role in the democratization of AI, yet their "open" nature introduces more avenues for malicious actors to misuse them for harmful purposes.