arXiv AI

The Fragility of Trigger-Tag Mechanisms for Misuse Detection in Open-Weight LLMs

arXiv AI
Oct 2

UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models

UniGuardian is a training‑free detector for large language models that jointly identifies prompt injection, backdoor, and adversarial attacks—collectively called Prompt Trigger Attacks (PTA). It measures how structured prompt perturbations shift the model’s output distribution and uses a single‑forward strategy to detect attacks while generating text in a shared batched forward pass. Experiments show that UniGuardian accurately and efficiently identifies trigger‑activated prompts in LLMs.

By Huawei Lin, Yingjie Lao, Tony Geng, Tan Yu, Weijie Zhao
arXiv Machine Learning
Jul 30

ToxScreen: Detecting Whether an LLM Has Been Poisoned

arXiv:2607. 26849v1 Announce Type: cross Abstract: As large language models (LLMs) are deployed in high-stakes domains, adversaries may poison training data to implant backdoors: hidden triggers that covertly manipulate model behavior at inference time.

By Anthony Hughes, Nicole Xing, Collin Francel, Andy Kim, Andrew Draganov
Hugging Face Trending Papers
Jul 29

ToxScreen: Detecting Whether an LLM Has Been Poisoned

As large language models (LLMs) are deployed in high-stakes domains, adversaries may poison training data to implant backdoors: hidden triggers that covertly manipulate model behavior at inference time. We ask whether a defender can recover such a trigger under realistic affordances, namely white-box access to the weights and knowledge of the behavior of concern, but no training data, no trusted reference model, no knowledge of the trigger, and no certainty that the model is poisoned.

arXiv AI
2d ago

Cancerous Backdoor Attack: Gradual Activation via Invisible Tokens and A Stage Controller

The paper introduces the Cancerous Backdoor Attack, a novel backdoor technique for large language models that activates gradually through multiple hidden stages rather than a single trigger-response. It uses invisible tokens to embed intermediate stages into the model’s reasoning process and a stage controller to monitor the attack’s progression during inference. Experiments on various reasoning datasets show that this method consistently corrupts model outputs over time and outperforms existing backdoor baselines.

By Tahmid Imtiaz Imu, Emama Nahid, Prodipta Sen Amartya, Ahsanul Jobayer, Yutong Song, Jiang Wu, Pengfei Zhang, Honghui Xu
arXiv Machine Learning
6d ago

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

The paper examines the vulnerability of open‑weight large language model safeguards that aim to block harmful usage by assuming new harmful behavior arises only through fine‑tuning. It demonstrates that simple, low‑cost attacks—abliteration and prefilling—can significantly increase attack success rates across multiple benchmarks, and proposes an abliteration‑resistant tuning (ART) method that mitigates these attacks. The study shows that safeguarding defenses must be evaluated against a broader range of attack strategies beyond fine‑tuning.

By Kevin Kuo, Virginia Smith, Chhavi Yadav
arXiv AI
Jun 16

MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks

arXiv:2602. 09222v2 Announce Type: replace-cross Abstract: Large language model (LLM) based web agents are increasingly deployed to automate complex online tasks by directly interacting with web sites and performing actions on users' behalf.

By Georgios Syros, Evan Rose, Brian Grinstead, Christoph Kerschbaumer, William Robertson, Cristina Nita-Rotaru, Alina Oprea