The study investigates whether large language models (LLMs) can reliably detect when their own responses have been manipulated by adversarial prefill attacks. Across ten instruction‑tuned LLMs ranging from 3B to 70B parameters and four safety benchmarks, none consistently recognized compromised outputs, with models claiming intent on prefilled responses at an average of 25.3%. The research identifies that introspective signals mainly arise from safety reasoning and refusal, and that training to improve introspection can paradoxically increase attack success, underscoring the fragility of LLM self‑reporting in safety contexts.
By Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim
arXiv:2608. 03201v1 Announce Type: new Abstract: Safety guards are widely used to filter harmful content and are typically trained via supervised fine-tuning on labeled prompt-response pairs.
By Yu Feng, Chunting Zang, Chen Shen, Rui Miao, Ge Teng, Weidong Cai, Jieping Ye
arXiv:2608. 11583v1 Announce Type: new Abstract: Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that refusal behavior may be concentrated in a smaller set of parameters.
By Mingyu Zong, Sampad Mohanty, Bhaskar Krishnamachari
The study investigates how lexical perturbations—such as keyboard noise, character swaps, and filler insertion—affect large language models (LLMs) on reasoning benchmarks. Four open-weight instruction-tuned models and frontier models were evaluated, revealing that character-level perturbations significantly reduce accuracy, especially on multi-step reasoning tasks, while filler insertion has minimal impact. The authors attribute this asymmetry to Attention Diversion, where fragmented subword tokenization draws disproportionate attention in middle and final transformer layers; they demonstrate that both token content and attention allocation are coupled, making it difficult for inference-time repair strategies to fully recover performance.
By Jiaqian Zhu, Yang Zhang, Junhua Ding, Xiaowei Yu
arXiv:2609.00051v1 Announce Type: cross
Abstract: Despite extensive alignment efforts, Large Language Models (LLMs) remain vulnerable to generating unsafe content under adversarial prompting, yet the...
By Kuan-Lin Chu, Chung-En Sun, Tsui-Wei Weng
The paper investigates how large language models balance helpfulness and safety by refusing harmful queries while responding to benign ones. It decomposes safety-tuning responses into a boilerplate refusal statement and a rationale, finding that the statement causes false refusals by relying on superficial cues. Training on rationales alone reduces false refusals without compromising safety performance, suggesting that fine‑grained safety supervision is essential for better alignment.
By Minji Kim, Hyounghun Kim
The paper introduces "cunning questions"—non‑safety prompts that contain misleading premises or subtle inconsistencies—to train large language models (LLMs) to scrutinize underlying intent and assumptions. Experiments show that incorporating these questions improves robustness against out‑of‑distribution jailbreak attacks and enhances subsequent safety fine‑tuning, achieving a new state‑of‑the‑art reduction in mean ASR from 17.40% to 15.05% across nine backbone–benchmark combinations. The authors argue that this training fosters vigilance, enabling models to prioritize safety judgments before engaging in harmful planning.
By Youjia Wang, Lin Xu, Yang Sun, Yuxiao Lu, Chengfang Fang, Jie Shi
The paper investigates a continuation-triggered jailbreak in large language models, showing that moving an instruction suffix can markedly boost jailbreak success. By performing mechanistic interpretability at the attention‑head level, the authors reveal that the jailbreak arises from a competition between the model’s natural continuation drive and safety defenses learned during alignment. They introduce Head Competition Steering (HCS), an inference‑time technique that exploits this competition to suppress harmful outputs and distill the approach into a student model for efficient safety improvements.
By Yonghong Deng, Zhen Yang, Ping Jian, Xinyue Zhang, Zhongbin Guo, Chengzhi Li, Junxi Yin
arXiv:2606. 31748v1 Announce Type: new Abstract: Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones.
By Taeyoun Kim, Aviral Kumar
The paper addresses the problem of over‑refusal in safety‑aligned large language models, where benign instructions are incorrectly rejected. It identifies that a small set of hypersensitive safety heads in transformer attention misfire on hard‑safe prompts, causing abnormal attention entanglement and high‑entropy routing conflicts that block necessary attention to target entities. To mitigate this, the authors propose Semantic Routing Calibration (SRC), a lightweight, training‑free inference framework that dynamically suppresses these hypersensitive heads and fuses logits from dual branches to restore trustworthy reasoning while preserving intrinsic safety performance.
By Zixuan Wang, Bingjie Zhang, He Zhao, Dandan Guo
arXiv:2609.22119v1 Announce Type: cross
Abstract: Evaluation awareness poses an unprecedented threat to model evaluation, but the mechanisms by which models detect it remain unknown. This study focus...
By Navraj Singh, Maheep Chaudhary
The paper identifies a vulnerability in large language models where harmful intent can be hidden within benign narratives, a phenomenon termed Semantic Camouflage. By examining latent activation patterns across several small language model families, the authors discover an "Intent Horizon"—a layer depth where harmful intent representations collapse. They propose Latent Intent Verification (LIV), a lightweight probing defense that detects harmful intent in early layers and outperforms existing guardrails on the PKU-SafeRLHF dataset.
By Md. Hasib Ur Rahman