arXiv Computation and Language
3d ago

Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model-Skill Co-Evolution

The paper introduces SGUID, a method that selects a compact subset of skills for distillation in large language models. It shows that fewer than 25% of retrieved skills are useful for on‑policy distillation and that distilling only the selected skills can match or surpass full‑bank distillation while using up to 11× fewer skills. SGUID also enables stable model‑skill co‑evolution by iteratively curating new skill banks from updated model rollouts.

By Yuhan Liu, Xiyao Ma, Zhongkai Sun, Xu Han, Chengyuan Ma, Benjamin Z. Yao, Chenlei Guo