arXiv AI

Model Hypnosis: Strong control of AI via additive subliminal effects

arXiv:2608. 16834v1 Announce Type: cross Abstract: We demonstrate that AI models are broadly susceptible to a phenomenon we call model hypnosis, in which individually weak and seemingly irrelevant cues in the prompt can be systematically combined to strongly control model behavior.

arXiv AI
5d ago

The First Few Tokens Matter: Understanding Chain-of-Thought Controllability

The paper investigates why chain-of-thought (CoT) reasoning in language models is hard to control, attributing the issue to an entropy collapse in the first few tokens after the opening thinking tag. By applying two inference-time interventions—prefilling a single token and banning frequent opening tokens with in-context examples—the authors significantly improve CoT controllability, especially for addition and style tasks. The study also distinguishes between addition/style and suppression controllability, revealing different underlying mechanisms and correlations with monitorability across models.

By Hugh Van Deventer, Dmitrii Troitskii, Laura Ruis
arXiv AI
Sep 16

Verbalizing Subliminal Learning Effects Using Text Optimization

The paper introduces SALVE, a method that uses text optimization to uncover and verbalize subliminal learning effects in language models. By optimizing a soft prompt and converting it into a legible text prompt, SALVE can reliably recover the teacher model’s hidden traits that are transmitted through a distillation dataset. The authors demonstrate SALVE’s effectiveness across various scenarios, including mixed datasets, biased teacher activation, and preference‑selected data, thereby providing a tool for detecting hidden influences in model training.

By Nathan Hu, Sanmi Koyejo, Christopher Potts