Hugging Face Trending Papers
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Aug 27
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
arXiv AI
Jul 21
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
arXiv Machine Learning
Jul 30
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
arXiv AI
Jul 28
TriSP: Tri-Signal Structured Pruning for Large Language Models
arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.
arXiv Machine Learning
Jun 18
Complementary Attention Head Pruning for Efficient Transformers
arXiv:2606. 19150v1 Announce Type: new Abstract: The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments.
arXiv AI
Jul 23
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.