arXiv:2607. 15450v1 Announce Type: cross Abstract: Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...
The paper studies weak-to-strong generalization (W2SG), where a student model trained on a weaker teacher’s labels surpasses the teacher on the target task. Using a Bregman divergence bias‑variance decomposition, it shows that the student‑teacher risk gap depends on their expected misfit, without requiring convexity of the student hypothesis class. For squared loss, a sufficient condition is that the student converges to the teacher’s posterior mean, achievable by enlarging the student; for cross‑entropy loss, reducing the student’s predictive entropy and using reverse cross‑entropy can promote W2SG, which is empirically validated.
The paper investigates how to choose the best quantized model from a family of compressed versions when target labels are scarce or unavailable. It finds that a simple rule based on minimum teacher distortion consistently selects the same eight‑bit, per‑channel, unclipped configuration, though this does not minimize empirical target cross‑entropy. The study also shows that confidence‑based estimators perform poorly in overconfident regimes, while output‑distribution estimators can outperform the teacher in some architectures, and that combining distortion with a supervised term can improve selection. Across 134 candidate families, teacher‑anchored selection reduces mean regret with very few labels, though the benefit diminishes after about 25 labels.
arXiv:2606. 00798v1 Announce Type: cross Abstract: Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student.
arXiv:2609.38011v1 Announce Type: new Abstract: Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downst...
arXiv:2603. 05691v3 Announce Type: replace Abstract: It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models.