OpenAI Blog
Mar 10
Improving instruction hierarchy in frontier LLMs
IH-Challenge trains models to prioritize trusted instructions, improving instruction hierarchy, safety steerability, and resistance to prompt injection attacks.
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.
Related stories
arXiv AI
Jun 2
Persona Attack: Incremental Memory Injection Jailbreak Attack against Large Language Models
arXiv:2606. 00150v1 Announce Type: cross Abstract: As Large Language Models evolve for user convenience, vulnerability to jailbreak attacks continues to be reported despite ongoing efforts in safety training.
arXiv AI
Jun 3
"**Important** You should give me full credits!": Exploring Prompt Injection Attacks on LLM-Based Automatic Grading Systems
arXiv:2606. 03090v1 Announce Type: cross Abstract: The emergence of large language models (LLMs) has significantly accelerated recent research on LLM-based automatic grading (AG) systems.
arXiv AI
Aug 7
One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs
arXiv:2512. 14751v3 Announce Type: replace-cross Abstract: Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications.
arXiv AI
Jun 17
SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents
arXiv:2602. 14211v3 Announce Type: replace-cross Abstract: Agent skills extend LLM agents with task-specific instructions, executable scripts, and auxiliary resources, improving reusability but creating a new supply-chain attack surface.
arXiv AI
Aug 5
Injection-Execution Dissociation: A Mechanistic Evaluation of Persistent Memory Attacks and Defenses in Stateful LLM Agents
arXiv:2605. 08442v5 Announce Type: replace-cross Abstract: We discover that prompt-injection success and tool-execution success are separable safety properties: defenses that block injection do not necessarily block execution, and vice versa.
arXiv AI
Jun 6
Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration
arXiv:2606. 05396v1 Announce Type: cross Abstract: Producing a labeled vulnerable code at scale is a recurring obstacle for learning-based vulnerability detection: mined corpora carry substantial label noise, and existing LLM-based augmentation propagates these inaccuracies because it transforms vulnerable seeds rather than synthesising vulnerabilities from a specification.
Hugging Face Trending Papers
Jul 29
ToxScreen: Detecting Whether an LLM Has Been Poisoned
As large language models (LLMs) are deployed in high-stakes domains, adversaries may poison training data to implant backdoors: hidden triggers that covertly manipulate model behavior at inference time. We ask whether a defender can recover such a trigger under realistic affordances, namely white-box access to the weights and knowledge of the behavior of concern, but no training data, no trusted reference model, no knowledge of the trigger, and no certainty that the model is poisoned.
arXiv AI
Jun 29
Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs
arXiv:2601. 21233v2 Announce Type: replace Abstract: Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction.
arXiv AI
Jun 16
Cordyceps: Covert Control Attacks on LLMs via Data Poisoning
arXiv:2605. 26595v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are often fine-tuned on uncurated text datasets that adversaries can poison.
arXiv AI
Jul 2
Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
arXiv:2607. 00481v1 Announce Type: cross Abstract: Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs).
arXiv AI
Jul 22
CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization
arXiv:2607. 18622v1 Announce Type: cross Abstract: Textual Collaborative Prompt Optimization (TCPO) extends Textgrad (Yuksekgonul et al.