← Back to all news
arXiv Machine Learning By Ravenor Davion, Nick Rui

Privileged Context as Drift in On-Policy Self-Distillation

Read the original on arXiv Machine Learning →

The paper investigates how different types of privileged context affect policy drift in on‑policy self‑distillation (OPSD) for language models. By varying content (demonstration, feedback, rephrase) and source (external, self‑generated with a verifier, self‑generated without a verifier) across nine combinations, the authors train Qwen2.5‑7B on three datasets and measure target‑task accuracy, prior‑task retention, reverse KL divergence, and parameter‑update geometry. Results show that changing content leads to a larger median KL range and more aligned parameter updates than changing source, indicating that privileged context significantly influences the direction and magnitude of policy updates.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 1

Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention

The paper investigates self‑distillation techniques for language models by systematically varying three key design choices: the source of rollout tokens (student vs. teacher), the teacher coupling strategy (frozen or exponential moving average), and the KL divergence direction (reverse or forward). Experiments on Qwen2.5‑7B and Ministral‑3‑3B across 1,200 adaptation runs reveal that rollout source mainly affects acquisition on contradictory tasks, teacher coupling most strongly influences acquisition across all tasks, and KL direction impacts retention differently depending on the model. A controlled theoretical model reproduces these empirical trends, offering a unified framework for understanding acquisition‑retention trade‑offs in self‑distillation.

By Luis Zuin, Alexis Huet, Dario Rossi, Zied Ben Houidi
llmsfine-tuningefficiency
More like this →
arXiv Computation and Language
Sep 18

What Does Privileged Information Add to On-Policy Self-Distillation?

The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.

By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
llmsefficiencybenchmarks
More like this →
arXiv AI
Aug 11

SR-OPSD: Self-Referenced On-Policy Self-Distillation

arXiv:2608. 09745v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards.

By Zhuo Sun, Entong Li, Yanlong Zhao, Xiaoyuan Cheng, Wenxuan Yuan, Kaiyu Li, Che Liu, Huihang Liu, Harrison Bo Hua Zhu, Li Zeng
llmsreinforcement-learningefficiencybenchmarks
More like this →