Hugging Face Trending Papers
Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
5d ago
Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs
The paper investigates a new attack method called output‑prefix attacks on reasoning LLMs, where an attacker prepends a malicious text to the model’s output, thereby conditioning all subsequent tokens on that prefix. The study systematically isolates the scratchpad reasoning channel as a vulnerable vector and compares three attack types—reasoning‑only, output‑prefix‑only, and combined reasoning‑plus‑output‑prefix—across both exposed and hidden reasoning models. Experiments on three 2026‑era frontier models (Gemini 3 Flash Preview, DeepSeek V4 Flash, and Claude Haiku 4.5) show that reasoning alone is largely ineffective, but adding a trivial output prefix can raise attack success rates to as high as 99% for some models, with contextual prefixes outperforming static ones and susceptibility varying by model.
arXiv Machine Learning
Sep 21
OverThink: Slowdown Attacks on Reasoning LLMs
The paper introduces OverThink, a slowdown attack that forces reasoning language models (RLMs) to produce many more reasoning tokens while still giving correct answers. By injecting decoy reasoning problems—such as Markov decision processes, language translation, or graphic comprehension—into the model’s context, attackers can dramatically increase token generation (up to 46× on SQuAD and 17× on coding agents). The study evaluates the attack on both proprietary and open-source RLMs across multiple datasets, explores multimodal and coding‑agent variants, and tests several defenses, concluding that defending against OverThink is challenging and that newer RLMs are even more vulnerable due to higher per‑token costs and increased reasoning token usage.
arXiv AI
Jun 9
RecurGuard: Runtime Monitoring for Reasoning-Token Consumption Attacks
arXiv:2606. 07968v1 Announce Type: cross Abstract: Reasoning-capable large language models can be induced to spend their generation budget on injected decoy tasks rather than answering the user's question, causing denial of service when no final answer is produced and denial of wallet when excess output tokens are billed.
arXiv AI
Aug 26
Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors
The paper introduces Semantic Overlays, a steering technique that adds non‑textual annotations to a language model’s input by applying learned adapters at specific prefill positions. These overlays create an out‑of‑band channel that encodes span identity and complex semantics, enabling the model to interpret marked text differently—such as rewriting code in a specified language or ignoring executable instructions. Experiments show that Semantic Overlays dramatically reduce prompt‑injection success rates while preserving model utility and readability of marked spans.
arXiv AI
Jun 26
HauntAttack: When Attack Follows Reasoning as a Shadow
arXiv:2506. 07031v5 Announce Type: replace-cross Abstract: Emerging Large Reasoning Models (LRMs) consistently excel in mathematical and reasoning tasks, showcasing remarkable capabilities.
arXiv Computation and Language
Sep 22
BreakFun: Jailbreaking LLMs via Object Instantiation under Simulated Code Execution
arXiv:2510.17904v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are widely used because they process structures, syntax and code well, but this same ability also makes them par...