The paper introduces Synthetic Self-Distillation (SynthSelf), a method that enables continual post‑training for small language models by outsourcing in‑distribution learning to a slightly larger aligner. SynthSelf combines off‑policy supervision and rejection sampling to provide high‑quality supervision while preserving the model’s original distribution. Experiments on four acquisition datasets and fourteen retention benchmarks show SynthSelf consistently outperforms six baseline methods across three small‑model scales, demonstrating its effectiveness for low‑latency, on‑device continual learning.
arXiv:2601. 19897v2 Announce Type: replace Abstract: Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models.
arXiv:2607. 01763v1 Announce Type: new Abstract: Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities.
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach.
arXiv:2609.24646v1 Announce Type: new Abstract: On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full d...
arXiv:2407. 13911v5 Announce Type: replace-cross Abstract: Prompt-based continual learning has shown strong performance in rehearsal-free class-incremental learning by adapting learnable prompts while freezing a pre-trained Vision Transformer (ViT) backbone.
HERD (Harness-Evolving Recursive Distillation) is a hybrid continual learning framework that combines harness-based non‑parametric methods with parametric distillation. It refines reusable skills in a harness while storing high‑residual examples in bounded memory, and applies drift‑constrained distillation with dual anchors to reduce forgetting. On the 25‑stage TRACE+17 task stream, HERD matches the best parametric baseline in new‑task learning while achieving the lowest forgetting and strongest retention of general capabilities under fixed memory and compute budgets.
The paper demonstrates that off‑policy merging, called grafting, outperforms on‑policy self‑distillation for continual learning. Grafting learns updates on an earlier donor checkpoint, scales the update, and optionally masks sensitive directions, thereby reducing interference with existing capabilities. Across various continual learning scenarios, grafting achieves better new‑task and old‑task performance without costly on‑policy sampling.
The paper argues that catastrophic forgetting and loss of plasticity alone cannot explain why naive sequential training underperforms offline joint training. It introduces data co-observation as a third factor, showing that observing training data together consistently improves performance across supervised and self-supervised settings. The study also reinterprets common continual learning methods, suggesting that memory replay’s success stems from restoring co-observation benefits rather than merely mitigating forgetting.
arXiv:2605.06597v3 Announce Type: replace-cross Abstract: Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. Howev...
arXiv:2607. 07847v1 Announce Type: new Abstract: As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn?
The paper investigates self‑distillation techniques for language models by systematically varying three key design choices: the source of rollout tokens (student vs. teacher), the teacher coupling strategy (frozen or exponential moving average), and the KL divergence direction (reverse or forward). Experiments on Qwen2.5‑7B and Ministral‑3‑3B across 1,200 adaptation runs reveal that rollout source mainly affects acquisition on contradictory tasks, teacher coupling most strongly influences acquisition across all tasks, and KL direction impacts retention differently depending on the model. A controlled theoretical model reproduces these empirical trends, offering a unified framework for understanding acquisition‑retention trade‑offs in self‑distillation.
arXiv:2404.07729v2 Announce Type: replace Abstract: Continual learning (CL) evaluates adaptability in learning solutions to retain knowledge. Our research addresses the challenge of catastrophic forg...