Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation

The paper introduces a new approach to low‑rank clone distillation that ensures the student model’s training targets exactly the weights it will use at inference. By redefining the training objective to cover the full deployed matrix—without changing the model’s shape, parameter count, or FLOPs—the authors recover previously unreachable linear degrees of freedom. This results in significant performance gains across multiple teacher models, achieving comparable or superior accuracy with fewer tokens and parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• fine-tuning
• efficiency
• benchmarks