Adaptive-Policy Distillation: Knowledge Injection under Information Asymmetry

The paper introduces Adaptive-Policy Distillation (APD), a method that improves knowledge injection from a context-conditioned teacher model into a student model by dynamically adjusting the distillation policy and loss function at each token. APD addresses two structural failures of On‑Policy Distillation (OPD): the reverse‑KL supervision’s weak influence on unlikely document tokens and the degradation of coverage for longer target sequences. Experiments on QASPER and BioASQ show that APD outperforms OPD by up to 54.6%, achieving state‑of‑the‑art F1 and MRR scores.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• efficiency
• benchmarks