arXiv AI
Jun 2
Subliminal Learning Is Steering Vector Distillation
arXiv:2606. 00995v1 Announce Type: new Abstract: Subliminal learning refers to a student language model acquiring a teacher's traits (e.
Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learning
The paper investigates subliminal learning (SL), where a student model learns downstream tasks by matching auxiliary outputs from a teacher without seeing task labels or original data. By deriving a chained cross‑task kernel, the authors link ghost‑output supervision to changes in task predictions via shared backbone representations. Their framework explains three empirical puzzles: a PSD transfer operator under shared initialization, a rank bottleneck set by ghost‑output dimensionality, and the superiority of high‑entropy synthetic inputs as broadband probes for maximizing kernel overlap.
Related stories
arXiv AI
Aug 7
Subliminal Learning is Non-Semantic Distillation
arXiv:2608. 05734v1 Announce Type: new Abstract: Subliminal Learning (SL) is a surprising type of generalization displayed by modern language models.
Hugging Face Trending Papers
Aug 6
Subliminal Learning is Non-Semantic Distillation
Subliminal Learning (SL) is a surprising type of generalization displayed by modern language models. It allows the transfer of a bias or behavior from a teacher model to a student by distilling from seemingly unrelated or random synthetic data from the teacher.
arXiv AI
Sep 16
Verbalizing Subliminal Learning Effects Using Text Optimization
The paper introduces SALVE, a method that uses text optimization to uncover and verbalize subliminal learning effects in language models. By optimizing a soft prompt and converting it into a legible text prompt, SALVE can reliably recover the teacher model’s hidden traits that are transmitted through a distillation dataset. The authors demonstrate SALVE’s effectiveness across various scenarios, including mixed datasets, biased teacher activation, and preference‑selected data, thereby providing a tool for detecting hidden influences in model training.
arXiv Machine Learning
Sep 1
Tracing distinguishability through transformer processing with stochastic LayerNorm
arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...
arXiv Machine Learning
Jul 28
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv AI
Sep 28
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
arXiv:2605. 08475v3 Announce Type: replace-cross Abstract: In this paper, we study in-context kernel ridge regression (KRR) with Gaussian kernels and show, both theoretically and empirically, that a standard softmax-attention transformer can approximate the KRR predictor during its forward pass.
arXiv Machine Learning
Oct 3
Localizing Transfer Between Memorization Tasks
The paper investigates how pre‑training on random input‑output mappings (memorization tasks) can transfer to downstream tasks. It discovers two unexpected patterns: equivalent transfer, where each pre‑training epoch saves roughly one fine‑tuning epoch, and non‑equivalent transfer, where pre‑training on a mismatched task can be more efficient than training directly on the downstream task. Ablation studies reveal that transfer consists of a trivial magnitude‑driven effect in the last layer and a non‑trivial structure‑driven effect linked to covariance in other layers.
arXiv Machine Learning
Jul 3
Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised Learning
arXiv:2503. 21796v2 Announce Type: replace-cross Abstract: Self-supervised learning has become an increasingly important paradigm in the domain of machine intelligence.
Hugging Face Trending Papers
Jul 15
Leveraging unlabelled data for generalizable neural population decoding
Robust and accurate neural decoders are integral to neurotechnologies such as brain-computer interfaces and closed-loop experiments. Recent work has shown that tokenizing neural data at the spike level facilitates multi-session pretraining and delivers state-of-the-art decoding performance.
arXiv Computer Vision
Sep 14
DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning
arXiv:2607.01630v2 Announce Type: replace Abstract: Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our...
arXiv Machine Learning
Sep 2
Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation
The paper investigates subliminal learning, where hidden traits from a teacher model are transferred to a student during distillation. It introduces trait‑direction drift as the underlying mechanism, showing that biased generation creates measurable preference gaps that accumulate into behavioral transfer during fine‑tuning. The authors propose probe‑space corridor regularization, a targeted defense that constrains drift along a calibrated trait direction, significantly reducing hidden‑trait transfer while maintaining task performance.