The paper proposes Self‑Distillation Fine‑Tuning (SDFT) as a method to recover performance in Large Language Models that has been degraded by catastrophic forgetting, quantization, or pruning. It shows that SDFT restores model capabilities by aligning the high‑dimensional manifold of the student model’s hidden layers with that of a teacher model, as measured by Centered Kernel Alignment (CKA). The authors provide both empirical evidence of strong correlation between manifold alignment and performance recovery and a theoretical explanation linking generative capability to the structure of these manifolds.
By Chi Liu, Xin Chen, Xu Zhou, Fangbo Tu, Srinivasan Manoharan
arXiv:2607. 15919v1 Announce Type: cross Abstract: Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data.
By Mohamed Amine Kina
arXiv:2412. 01282v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks.
By Qianhan Feng, Wenshuo Li, Tong Lin, Xinghao Chen
arXiv:2603. 14830v3 Announce Type: replace Abstract: Dataset distillation, a training-aware data compression technique, has recently attracted increasing attention as an effective tool for mitigating costs of optimization and data storage.
By Yuri Kinoshita, Naoki Nishikawa, Taro Toyoizumi
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.
arXiv:2609.40047v1 Announce Type: cross
Abstract: Gromov-Wasserstein multidimensional scaling (GW-MDS) learns low-dimensional representations from relational data but remains transductive, providing...
By Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro Cavalcante