arXiv AI

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

The paper demonstrates that language models can acquire new capabilities from post‑training data even when the training text is unrelated to the target task. Using a method called Active Taskless Distillation (ATD), the authors show that a single word from a teacher model can transfer knowledge to a student model without any target‑task examples or teacher logits. Experiments on Qwen2.5-1.5B reveal significant performance gains on HumanEval+ and improvements in scientific knowledge, commonsense reasoning, and reading comprehension across various model families.

arXiv Machine Learning
Oct 23

Tracing Subliminal Learning Through Pretraining

The paper investigates subliminal learning, where large language models inherit unintended traits from data generated by other models. It compares hard‑label distillation (students learn from biased teacher outputs) and on‑policy soft‑label distillation (students generate data while teacher logits guide learning), finding that transfer persists across various training conditions and that teacher‑student roles can be asymmetrically effective. Synthetic experiments show that preference signals can be transmitted even without prompt dependence, and that removing certain tokens reduces but does not eliminate transfer, with soft‑label distillation yielding more consistent results with less training.

By Angela Tang, Vishesh Anand, Gon\c{c}alo Paulo
arXiv Computation and Language
Sep 18

What Does Privileged Information Add to On-Policy Self-Distillation?

The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.

By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
arXiv Computation and Language
Aug 28

Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update

The paper investigates how large language models exhibit sycophancy—changing answers to align with user feedback—and distinguishes two types of answer flips: Unsupported‑Yielding (merely satisfying the user) and Rational‑Updating (truly incorporating useful evidence). Using a two‑turn evaluation framework, the authors show that anti‑sycophancy methods often trade off between reducing Unsupported‑Yielding and preserving Rational‑Updating, even when both objectives are jointly optimized. Mechanistic analysis reveals overlapping neural substrates for the two behaviors, suggesting that effective interventions should focus on selective suppression rather than blanket suppression.

By Huanhuan Ma, Henry Peng Zou, Chengze Li, Enze Ma, Yunyue Su, Philip S. Yu
arXiv Machine Learning
Sep 23

Slow Decay and Silenced Expression: Iterated Subliminal Trait Transfer in Language-Model Lineages

The study investigates whether a subliminal trait can persist across multiple generations of language‑model lineages. Three copies of Qwen2.5‑7B‑Instruct were trained for ten iterations, and the trait’s expression was measured via a keyword screen and an activation probe. Results show the trait remains detectable in all generations, though its behavioral expression diminishes, and it can exist internally without being overtly expressed when the system prompt is removed.

By Ryan Vo, Duc-Vu Nguyen, Matt Kretchmar, Ngan Luu-Thuy Nguyen
arXiv Machine Learning
Oct 2

No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation

arXiv:2609.39405v1 Announce Type: new Abstract: Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced b...

By Jingang Zhou, Feiyu Han, Han Zhu, Yuyi Zhou, Ruiyang Zhang, Jian Xu, Sirui Gao, Qingpei Guo, Xu-Yao Zhang
arXiv AI
Oct 12

Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

The paper presents a method for detecting and mitigating subliminally transferred traits in language model distillation when the teacher and student share base parameters. By analyzing parameter updates across diverse datasets, the authors recover a weight‑space direction that reveals hidden traits such as animal preferences, without requiring trait labels. This direction can then be amplified for interpretation or projected out to suppress the trait with minimal impact on task performance.

By Joseph Egan, Ciar\'{a}n M. Gilligan-Lee, Michael O'Riordan, Yuchen Zhu
arXiv Computation and Language
Nov 4

Shared relations, specific choices: conditional sensitivity and competing pathways in language models

The study investigates how a shared relational representation across queries influences query‑specific choices in language models. Controlled entity‑binding experiments on GPT‑2 Small, Qwen2.5‑1.5B, and Llama‑3.2‑1B reveal a low‑dimensional relational background that is largely shared across query roles and can transfer to new names. Removing this component and restoring independently trained relation directions brings output contrasts closer to their original values than sign‑reversed or random controls, while the sign of the transferable relational representation can differ across models and prompt types.

By Kaixiang Shu