The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
By Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
By Xin Yu, Cong Xie, Xunmei Liu, Tiantian Fan, Lingzhou Xue, Zhi Zhang
arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.
By Manel Kara laoua, Soumia Bouyahiaoui, Aicha Boutorh
arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.
By Yaniv Livertovsky, Shahar Somin, Gonen Singer
arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.
By Ashutosh Tripathi, Surya Deep Singh, Pranab Sahoo, Sriparna Saha
MoARa introduces a module-aware rank allocation strategy and a block-wise magnitude-direction decomposition to improve low-rank gradient projection for large language model pre‑training. By profiling Transformer modules and tailoring projection ranks, it reduces the number of steps and wall‑clock time needed to reach target perplexity. Experiments on Llama, Qwen, and DeepSeek models show up to 41.7% fewer steps and 37.1% less training time with minimal memory overhead.
By Keunyoung Kim, Nojun Kwak
arXiv:2602. 12429v2 Announce Type: replace Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges.
By Paul Janson, Edouard Oyallon, Eugene Belilovsky
Deploying Large Language Models (LLMs) in practice incurs substantial memory and computational costs. Post-training pruning (PTP) is an effective approach to reducing these costs by removing weights without additional training.
arXiv:2606. 01544v1 Announce Type: new Abstract: Deploying Large Language Models (LLMs) in practice incurs substantial memory and computational costs.
By Cheonjun Park
arXiv:2607. 22251v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules.
By Wei Zhang, Xinwu Liu, Yihang Cheng
arXiv:2608. 06901v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments.
By Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung