← Back to all news
arXiv AI By Zixuan Weng, Haoqi He, Weifei Jin, Yue Huang, Yang Bai

Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs

Read the original on arXiv AI →

The paper investigates how safety‑aligned large language models (LLMs) can still be tricked by jailbreaks or over‑refuse benign requests. By analyzing residual‑stream updates in decoder‑only Transformers, the authors find that harmfulness is more readily decoded from attention updates—especially in early layers—while the MLP component is primarily responsible for writing refusal responses. Post‑training enhances MLP‑driven refusal writing without improving harmfulness decodability, and causal interventions confirm that downstream MLP updates are crucial for the refusal shift caused by upstream attention changes. Building on these insights, the authors propose Sentinel, a system that combines an attention‑derived perception score with an MLP‑derived decision score to better detect jailbreaks and over‑refusals across multiple models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 2

Can LLMs Reliably Self-Report Adversarial Prefills, and How?

The study investigates whether large language models (LLMs) can reliably detect when their own responses have been manipulated by adversarial prefill attacks. Across ten instruction‑tuned LLMs ranging from 3B to 70B parameters and four safety benchmarks, none consistently recognized compromised outputs, with models claiming intent on prefilled responses at an average of 25.3%. The research identifies that introspective signals mainly arise from safety reasoning and refusal, and that training to improve introspection can paradoxically increase attack success, underscoring the fragility of LLM self‑reporting in safety contexts.

By Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim
llmsbenchmarkssafety
More like this →
arXiv AI
Aug 25

Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion

The study investigates how lexical perturbations—such as keyboard noise, character swaps, and filler insertion—affect large language models (LLMs) on reasoning benchmarks. Four open-weight instruction-tuned models and frontier models were evaluated, revealing that character-level perturbations significantly reduce accuracy, especially on multi-step reasoning tasks, while filler insertion has minimal impact. The authors attribute this asymmetry to Attention Diversion, where fragmented subword tokenization draws disproportionate attention in middle and final transformer layers; they demonstrate that both token content and attention allocation are coupled, making it difficult for inference-time repair strategies to fully recover performance.

By Jiaqian Zhu, Yang Zhang, Junhua Ding, Xiaowei Yu
llmsnlpbenchmarks
More like this →
arXiv AI
Sep 7

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

The paper investigates how large language models balance helpfulness and safety by refusing harmful queries while responding to benign ones. It decomposes safety-tuning responses into a boilerplate refusal statement and a rationale, finding that the statement causes false refusals by relying on superficial cues. Training on rationales alone reduces false refusals without compromising safety performance, suggesting that fine‑grained safety supervision is essential for better alignment.

By Minji Kim, Hyounghun Kim
llmsagentssafety
More like this →