arXiv AI

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

arXiv AI

Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 1

CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

arXiv:2606. 31796v1 Announce Type: cross Abstract: We study three complementary techniques for training compute-efficient language models.

By Dohyeon Kwon, Youngjin Park

llms

arXiv AI

Jul 20

KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.

By Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, Jinan Xu

llms nlp efficiency

arXiv Computation and Language

Oct 27

Improving the Performance of Compressed Transformers with Post-Hoc Embedding Alignment

arXiv:2602.14635v2 Announce Type: replace-cross Abstract: Model compression is necessary for deployment of transformer models in resource-constrained environments. However, compression reduces the pe...

By Rohit Raj Rai, Abhishek Dhaka, Amit Awekar

llms rag nlp fine-tuning efficiency safety

arXiv AI

Jul 24

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).

By Ning Yang, Hengyu Zhong, Wentao Wang, Baoliang Tian, Haijun Zhang, Jun Wang

llms rag efficiency benchmarks

arXiv AI

Jul 17

In-Place Tokenizer Expansion for Pre-trained LLMs

arXiv:2607. 15232v1 Announce Type: cross Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time.

By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner

llms rag nlp

arXiv AI

Jun 29

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

arXiv:2601. 22709v5 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.

By Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.

By Jianlu Shen, Fu Feng, Yucheng Xie, Jiaqi Lv, Xin Geng

fine-tuning benchmarks

arXiv AI

Oct 2

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads

arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...

By Prachi Badarayani, Aidan Jay, Chenghui Zhou, Dayquan Julienne, Yuan Gao, Tianwei Chen, George Zerveas, Ishmam Zabir, Xiren Zhou, Chris Quirk, Xia Song

llms efficiency benchmarks

arXiv Machine Learning

Jun 9

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li

llms nlp fine-tuning efficiency benchmarks

arXiv AI

Oct 7

Learning to Decide, Not to Reason: Parameter-Efficient Decision Operators via Low-Rank Activation Steering

arXiv:2610. 06950v1 Announce Type: cross Abstract: Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline.

By Ran Li, Lei Chen

llms reinforcement-learning

arXiv AI

Aug 11

Length-MAX Tokenizer for Language Models

arXiv:2511. 20849v2 Announce Type: replace-cross Abstract: We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference.

By Dong Dong, Weijie Su

llms rag nlp efficiency

Related stories