arXiv AI
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 21
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
arXiv AI
Jul 1
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
arXiv:2606. 31796v1 Announce Type: cross Abstract: We study three complementary techniques for training compute-efficient language models.
arXiv AI
Jul 20
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.
arXiv AI
Jul 24
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
arXiv AI
Jul 17
In-Place Tokenizer Expansion for Pre-trained LLMs
arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.
arXiv AI
Jun 29
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.