← Back to all news
arXiv AI By Yunbo Long, WenJie Chen, Jiaquan Zhang, Guangya Hao, Zihang Zeng, Pengze Li, Xi Chen

SPECTRUM: Proximal Spectral Modulation for Looped Self-Distillation

Read the original on arXiv AI →

The paper introduces SPECTRUM, a method for Loop Self‑Distillation that repeatedly generates and learns from a model’s own outputs while maintaining a fixed information budget. By re‑estimating loss‑sensitive key/value geometry from a reference anchor and applying full‑rank proximal spectral modulation, SPECTRUM preserves a high proportion of the initial model’s correct implementation diversity across multiple rounds of training. Experiments on MBPP, HumanEval+, and APPS Intro show that SPECTRUM retains more correct‑solution breadth than vanilla self‑distillation and subspace‑projection controls, and the resulting student model achieves higher matched‑correct richness without additional training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 1

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

The paper investigates self‑distillation techniques for language models by systematically varying three key design choices: the source of rollout tokens (student vs. teacher), the teacher coupling strategy (frozen or exponential moving average), and the KL divergence direction (reverse or forward). Experiments on Qwen2.5‑7B and Ministral‑3‑3B across 1,200 adaptation runs reveal that rollout source mainly affects acquisition on contradictory tasks, teacher coupling most strongly influences acquisition across all tasks, and KL direction impacts retention differently depending on the model. A controlled theoretical model reproduces these empirical trends, offering a unified framework for understanding acquisition‑retention trade‑offs in self‑distillation.

By Luis Zuin, Alexis Huet, Dario Rossi, Zied Ben Houidi
llmsfine-tuningefficiency
More like this →
arXiv AI
3d ago

UniSD: Towards a Unified Self-Distillation Framework for Large Language Models

UniSD presents a unified self‑distillation framework for large language models, integrating mechanisms such as multi‑teacher agreement, EMA teacher stabilization, token‑level contrastive learning, feature matching, and divergence clipping to address supervision reliability, representation alignment, and training stability. Evaluated across six benchmarks and multiple model families, UniSD identifies when self‑distillation outperforms static imitation and which components contribute most to performance gains. The resulting UniSDfull pipeline achieves the best overall results, improving the base model by +5.4 points and surpassing the strongest baseline by +2.8 points, demonstrating self‑distillation as a practical, steerable method for efficient LLM adaptation without stronger external teachers.

By Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Chao Zhang, Josiah Hester, Jindong Wang, Srijan Kumar
llmsefficiencybenchmarkssafety
More like this →
arXiv AI
Oct 12

Continually Post-training Small Models via Synthetic Self-distillation

The paper introduces Synthetic Self-Distillation (SynthSelf), a method that enables continual post‑training for small language models by outsourcing in‑distribution learning to a slightly larger aligner. SynthSelf combines off‑policy supervision and rejection sampling to provide high‑quality supervision while preserving the model’s original distribution. Experiments on four acquisition datasets and fourteen retention benchmarks show SynthSelf consistently outperforms six baseline methods across three small‑model scales, demonstrating its effectiveness for low‑latency, on‑device continual learning.

By Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee
llmsfine-tuningefficiencybenchmarks
More like this →