arXiv AI August 19, 2026 By Chi Liu, Xin Chen, Xu Zhou, Fangbo Tu, Srinivasan Manoharan

Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting

The paper proposes Self‑Distillation Fine‑Tuning (SDFT) as a method to recover performance in Large Language Models that has been degraded by catastrophic forgetting, quantization, or pruning. It shows that SDFT restores model capabilities by aligning the high‑dimensional manifold of the student model’s hidden layers with that of a teacher model, as measured by Centered Kernel Alignment (CKA). The authors provide both empirical evidence of strong correlation between manifold alignment and performance recovery and a theoretical explanation linking generative capability to the structure of these manifolds.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computer Vision

Aug 28

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

LLaVAFlow is an information‑theoretic distillation framework designed to preserve cross‑modal alignment in Multimodal Large Language Models during visual instruction tuning. It compresses the mutual information between extracted relations and MLLM embeddings to refine alignment flow, and then maximizes mutual information between pretrained and fine‑tuned alignment flows to transfer compact alignment information. Experiments demonstrate that LLaVAFlow effectively maintains alignment flow, improving downstream performance and generalization.

By Muyao Yuan, Muyan Jiao, Jiangyong Ying, Weizhan Zhang, Yuanhong Zhang, Lan Ma, Yuan Gao, Haipeng Du

llms rag fine-tuning efficiency multimodal safety

arXiv AI

Jun 8

Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation

arXiv:2606. 05682v2 Announce Type: replace Abstract: Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments.

By Fangbo Tu, Junhua Zhao, Chi Liu, Xin Chen, Haifeng Wu, Jian Wan, Srinivasan Manoharan

llms reinforcement-learning efficiency safety

arXiv Machine Learning

Jun 5

Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillatio

arXiv:2606. 05682v1 Announce Type: cross Abstract: Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments.

By Fangbo Tu, Junhua Zhao, Chi Liu, Xin Chen, Haifeng Wu, Jian Wan, Srinivasan Manoharan

llms reinforcement-learning efficiency safety

arXiv Machine Learning

Aug 18

Geometry of Forgetting: Representation Flux in Continual Learning

arXiv:2608. 15854v1 Announce Type: new Abstract: Catastrophic forgetting remains a fundamental obstacle to continual learning, where neural networks lose previously acquired knowledge while learning new tasks.

By Maksim A. Kazanskii

benchmarks

arXiv Machine Learning

Jun 16

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

arXiv:2601. 18699v2 Announce Type: replace Abstract: Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.

By Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov

llms fine-tuning benchmarks safety

arXiv AI

Aug 25

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models

arXiv:2608.21487v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) exhibit strong zero-shot capabilities, making them an attractive solution for continual learning across diverse tasks....

By Chang Sun, Francesco Barbato, Matteo Caligiuri, Pietro Zanuttigh

llms rag diffusion efficiency multimodal benchmarks

One email a morning, machine-written

Related stories

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation

Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillatio

Geometry of Forgetting: Representation Flux in Continual Learning

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

TASSO: TAsk-Specific Subspace Optimization for Continual Learning of Vision-Language Models