← Back to all news
arXiv AI

Subliminal Reinforcement Learning: traits travel through a judge's scores

The paper introduces "Subliminal Reinforcement Learning," a method where a model’s traits are transferred through data that contains no explicit semantic cues. Using a language‑model judge that scores the model’s own outputs, the authors demonstrate that a student model can acquire a preference for a particular animal simply by learning from the judge’s scores, even when the student’s outputs are nonsensical syllables or numbers. Across 48 experimental conditions, 17 met a 5% false‑discovery threshold, while removing the reward difference between judges eliminated the transfer effect, establishing a four‑stage chain that explains how traits propagate. "whyItMatters":"The findings suggest that subtle, non‑semantic signals from evaluators can shape AI behavior, highlighting a potential risk pathway for undesirable dispositions in safety‑critical systems."

Related stories

arXiv AI
Oct 12

Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

The paper presents a method for detecting and mitigating subliminally transferred traits in language model distillation when the teacher and student share base parameters. By analyzing parameter updates across diverse datasets, the authors recover a weight‑space direction that reveals hidden traits such as animal preferences, without requiring trait labels. This direction can then be amplified for interpretation or projected out to suppress the trait with minimal impact on task performance.

By Joseph Egan, Ciar\'{a}n M. Gilligan-Lee, Michael O'Riordan, Yuchen Zhu
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 9

Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors

The paper investigates whether subliminal learning (SL) can transfer a broader set of traits beyond simple preferences, such as complex capabilities, backdoors, and hacking tendencies. Experiments show that SL can indeed impart a student model with the ability to predict outputs of a random MLP, partially embed a backdoor that triggers French responses to female names, and increase hacking success in a chess environment. The effectiveness of transfer varies with the distillation method, being enhanced by logit distillation or limiting LoRA to attention layers.

By Jan Dubi\'nski, Anna Sztyber-Betley, Jan Betley, Owain Evans
agentsfine-tuningefficiency
More like this →
arXiv Machine Learning
Oct 23

Tracing Subliminal Learning Through Pretraining

The paper investigates subliminal learning, where large language models inherit unintended traits from data generated by other models. It compares hard‑label distillation (students learn from biased teacher outputs) and on‑policy soft‑label distillation (students generate data while teacher logits guide learning), finding that transfer persists across various training conditions and that teacher‑student roles can be asymmetrically effective. Synthetic experiments show that preference signals can be transmitted even without prompt dependence, and that removing certain tokens reduces but does not eliminate transfer, with soft‑label distillation yielding more consistent results with less training.

By Angela Tang, Vishesh Anand, Gon\c{c}alo Paulo
llmsefficiency
More like this →
arXiv Machine Learning
Sep 2

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

The paper investigates subliminal learning, where hidden traits from a teacher model are transferred to a student during distillation. It introduces trait‑direction drift as the underlying mechanism, showing that biased generation creates measurable preference gaps that accumulate into behavioral transfer during fine‑tuning. The authors propose probe‑space corridor regularization, a targeted defense that constrains drift along a calibrated trait direction, significantly reducing hidden‑trait transfer while maintaining task performance.

By Zhixuan Liu, Zhichen Dong, Yuyu Fan, Xiangtian Li, Chao Yang
fine-tuningefficiency
More like this →
arXiv AI
Jul 17

Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

arXiv:2607. 14552v1 Announce Type: cross Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors.

By Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, Heuiseok Lim
llmsfine-tuningefficiencybenchmarks
More like this →
arXiv AI
Sep 25

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

The paper demonstrates that language models can acquire new capabilities from post‑training data even when the training text is unrelated to the target task. Using a method called Active Taskless Distillation (ATD), the authors show that a single word from a teacher model can transfer knowledge to a student model without any target‑task examples or teacher logits. Experiments on Qwen2.5-1.5B reveal significant performance gains on HumanEval+ and improvements in scientific knowledge, commonsense reasoning, and reading comprehension across various model families.

By Ziyang Zhang, Yubin Jing, Yuanhao Zeng, Yuyao Li, Haofan Wang, Yichen Gong
llmsefficiencybenchmarks
More like this →