Distillation as Probability Transport: Routed On-Policy Distillation

The paper introduces RouteOPD, a new on‑policy distillation method that treats teacher‑student disagreement as a probability transport problem. By decomposing disagreement into excess sources and deficit destinations, RouteOPD pairs them explicitly and optimizes pairwise log‑odds toward targets derived from a bounded teacher potential. Experiments on four teacher‑student configurations and four mathematical‑reasoning benchmarks show that RouteOPD consistently outperforms sampled reverse‑KL OPD, achieving higher routing fidelity and lower background leakage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• efficiency
• benchmarks