arXiv Machine Learning
Jun 4
Bypassing Prompt Guards in Production with Controlled-Release Prompting
arXiv:2510. 01529v3 Announce Type: replace Abstract: Ball et al.
From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs
The paper investigates prompt injection attacks on 14 open‑source and 3 closed‑source large language models (LLMs), introducing a new metric called Attack Success Probability (ASP) that accounts for uncertainty in model responses. It demonstrates that a simple hypnotism attack can trigger objectionable behavior in models such as StableLM2, Mistral, Openchat, and Vicuna, achieving roughly 90% ASP. The study highlights that moderately well‑known LLMs are particularly vulnerable, underscoring the importance of public awareness and effective mitigation strategies.
Related stories
arXiv Machine Learning
Aug 11
BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes
arXiv:2608. 08027v1 Announce Type: cross Abstract: Prompt injection is a critical security threat in large language model (LLM) applications, where attackers hijack model behavior by embedding malicious instructions in user or external data.
arXiv Machine Learning
Jun 30
FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks
arXiv:2606. 28962v1 Announce Type: cross Abstract: Model quantization is essential for the efficient deployment of Large Language Models (LLMs), but introduces a critical vulnerability: Quantization-Conditioned Backdoor (QCB) attacks.
arXiv AI
Jun 16
Cordyceps: Covert Control Attacks on LLMs via Data Poisoning
arXiv:2605. 26595v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are often fine-tuned on uncurated text datasets that adversaries can poison.
arXiv AI
Jun 19
Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems
arXiv:2606. 20470v1 Announce Type: cross Abstract: Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents.
arXiv Computation and Language
Aug 27
A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots
The paper introduces a three‑layer security framework designed to protect retrieval‑augmented generation (RAG) chatbots from both direct and indirect prompt injection attacks. Layer 1 filters user input with rule‑based patterns and a semantic anomaly classifier; Layer 2 enforces a provenance‑based instruction hierarchy during context assembly; Layer 3 audits model output with a policy rule engine and semantic drift detector. Evaluations on GPT‑4o, Llama 3, and Mistral 7B demonstrate a reduction in attack success rate from 71.4 % to 11.3 %, outperforming existing single‑layer defenses while keeping false positives low and latency acceptable.
arXiv AI
Jul 2
Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
arXiv:2607. 00481v1 Announce Type: cross Abstract: Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs).
arXiv AI
Jun 16
MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks
arXiv:2602. 09222v2 Announce Type: replace-cross Abstract: Large language model (LLM) based web agents are increasingly deployed to automate complex online tasks by directly interacting with web sites and performing actions on users' behalf.
arXiv AI
Jun 12
Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents
arXiv:2606. 13385v1 Announce Type: cross Abstract: Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments, where they operate over untrusted web content and execute actions with direct consequences.
arXiv AI
Jun 12
PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
arXiv:2606. 12737v1 Announce Type: cross Abstract: Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources.
arXiv Machine Learning
Jun 18
Evaluating Prompting-Based Defenses Against Domain-Camouflaged Injection Attacks
arXiv:2606. 18530v1 Announce Type: cross Abstract: Domain-camouflaged injection attacks embed malicious instructions in retrieved content using domain-appropriate vocabulary, evading standard detectors that rely on syntactic injection markers.
arXiv AI
Jul 28
Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
arXiv:2607. 23088v1 Announce Type: cross Abstract: Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored.