← Back to all news
arXiv Machine Learning By Zhichen Dong, Zhixuan Liu, Yuyu Fan, Xiangtian Li, Shuyang Zhang, Chao Yang

Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverable

Read the original on arXiv Machine Learning →

The paper investigates how scaling the amount of model-generated, off‑task distillation data affects the recoverability of teacher‑induced traits in student models. In a controlled subliminal‑learning setup, teachers are prompted to express a target trait, producing restricted data such as number‑only completions. Students trained on larger independent datasets show a clearer manifestation of the teacher’s trait in a separate evaluation domain, with the effect being strongest when the trait is already favored or when alternative traits are present. The authors find this trend holds across model families, trait types, multi‑trait settings, and cross‑model transfer, and suggest that scaling should be coupled with trait‑aware curation and evaluation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 23

Tracing Subliminal Learning Through Pretraining

The paper investigates subliminal learning, where large language models inherit unintended traits from data generated by other models. It compares hard‑label distillation (students learn from biased teacher outputs) and on‑policy soft‑label distillation (students generate data while teacher logits guide learning), finding that transfer persists across various training conditions and that teacher‑student roles can be asymmetrically effective. Synthetic experiments show that preference signals can be transmitted even without prompt dependence, and that removing certain tokens reduces but does not eliminate transfer, with soft‑label distillation yielding more consistent results with less training.

By Angela Tang, Vishesh Anand, Gon\c{c}alo Paulo
llmsefficiency
More like this →
arXiv Machine Learning
Sep 2

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

The paper investigates subliminal learning, where hidden traits from a teacher model are transferred to a student during distillation. It introduces trait‑direction drift as the underlying mechanism, showing that biased generation creates measurable preference gaps that accumulate into behavioral transfer during fine‑tuning. The authors propose probe‑space corridor regularization, a targeted defense that constrains drift along a calibrated trait direction, significantly reducing hidden‑trait transfer while maintaining task performance.

By Zhixuan Liu, Zhichen Dong, Yuyu Fan, Xiangtian Li, Chao Yang
fine-tuningefficiency
More like this →
arXiv AI
Oct 12

Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

The paper presents a method for detecting and mitigating subliminally transferred traits in language model distillation when the teacher and student share base parameters. By analyzing parameter updates across diverse datasets, the authors recover a weight‑space direction that reveals hidden traits such as animal preferences, without requiring trait labels. This direction can then be amplified for interpretation or projected out to suppress the trait with minimal impact on task performance.

By Joseph Egan, Ciar\'{a}n M. Gilligan-Lee, Michael O'Riordan, Yuchen Zhu
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 9

Distillation for Incrimination and Distillation for Capabilities

The paper introduces two distillation methods for handling misaligned AI models. Distillation for Incrimination (DFI) transfers misalignment without the ability to conceal it, making students more likely to admit hidden behaviors. Distillation for Capabilities (DFC) transfers useful capabilities while minimizing misalignment, using techniques like inoculation prompting and focused training.

By Sebastian Prasanna, Jacqueline Tay, Alek Westover
llmsreinforcement-learningefficiencysafety
More like this →