← Back to all news
arXiv Computation and Language By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua

What Does Privileged Information Add to On-Policy Self-Distillation?

Read the original on arXiv Computation and Language →

The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 20

Rethinking Privileged Information in On-Policy Self-Distillation

The paper investigates on‑policy self‑distillation (OPSD), where a student model learns from its own outputs using token‑level supervision conditioned on privileged reference information. Experiments with Qwen3 models on science and mathematics datasets show that the correct reference does not consistently improve performance; students can improve without it, and solutions from other problems sometimes outperform the correct reference. The study finds that student predictions align more closely with the base model’s reasoning than with the reference supervision, and that alignment alone does not reliably predict performance gains.

By Samyak Shrestha, Alexander Tessier
efficiencybenchmarkssafety
More like this →