Continually Post-training Small Models via Synthetic Self-distillation

The paper introduces Synthetic Self-Distillation (SynthSelf), a method that enables continual post‑training for small language models by outsourcing in‑distribution learning to a slightly larger aligner. SynthSelf combines off‑policy supervision and rejection sampling to provide high‑quality supervision while preserving the model’s original distribution. Experiments on four acquisition datasets and fourteen retention benchmarks show SynthSelf consistently outperforms six baseline methods across three small‑model scales, demonstrating its effectiveness for low‑latency, on‑device continual learning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• fine-tuning
• efficiency
• benchmarks