← Back to all news
arXiv Machine Learning By Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei, Anisha Gunjal, Mohit Raghavendra, Lu Wang, Yunzhong He, Anas Mahmoud

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

Read the original on arXiv Machine Learning →

The paper studies how different configurations of privileged information (PI) affect on‑policy self‑distillation (OPSD) for large language models. By evaluating 16 PI setups across three agentic benchmarks and multiple student models, it finds that PI effectiveness varies with content, generation source, and model scale, with information‑dense trajectories and stronger generators yielding larger gains. The authors also propose KL‑based metrics, particularly Δd−_err, to predict PI utility before training, showing that selecting PI with this metric improves post‑training performance and remains beneficial when combined with GRPO.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 6

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

arXiv:2608. 04794v1 Announce Type: new Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it.

By Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi
agentsnlpreinforcement-learningefficiencysafety
More like this →
arXiv AI
Sep 25

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

The paper introduces Privileged Self-Practice (PSP), a method that retains privileged information (PI) in the prompt rather than the loss during on‑policy self‑distillation for multi‑turn agents. PSP injects short per‑task instructions from an analyzer model when rollouts fail, sampling again with the instruction in context and training with the unchanged GRPO objective. Experiments on AppWorld and SWE‑bench Verified show PSP consistently outperforms plain GRPO, boosting task‑goal completion by up to 65% and resolved rate by up to 61% across three student models.

By Xingyu Su, Abhishek Kumar, Qing Ping, Youzhi Luo, Jonathan Buck, Zach Zhang, Subramanian Chidambaram, Vinayak Arannil
llmsagentsreinforcement-learningefficiency
More like this →
Hugging Face Trending Papers
Sep 24

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

The paper examines on‑policy self‑distillation (OPSD) for multi‑turn agents, showing that using privileged information (PI) in the loss can make agents appear confident yet underperform plain RL, sometimes worse than the untrained base model. To address this, the authors propose Privileged Self‑Practice (PSP), which keeps PI in the prompt and uses it only during sampling, not in the loss. PSP consistently outperforms plain GRPO across AppWorld and SWE‑bench Verified, improving task‑goal completion by up to 65% and resolved rate by up to 61%.

llmsagentsreinforcement-learningefficiency
More like this →
arXiv AI
Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
llmsefficiencybenchmarks
More like this →