Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

Latent-MOPD is a new on‑policy distillation method that allows a single large language model student to learn from multiple specialist teachers by using both the teachers’ output distributions and their hidden state representations. The approach selects late‑layer targets based on teacher‑student relationships, bridges hidden width differences with a shared projection, and groups updates by domain, enabling gradual shift from representation to token supervision. Experiments show that Latent‑MOPD outperforms token‑only, representation‑only, and uniform‑averaging baselines across nine benchmarks in math, code, and logic, and even surpasses the best individual teacher on most tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• efficiency
• benchmarks