Hugging Face Trending Papers
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Aug 27
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
arXiv AI
Jul 21
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
arXiv Machine Learning
Jul 30
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
arXiv Machine Learning
Oct 15
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
arXiv:2311.04902v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurt...
arXiv AI
Jul 28
TriSP: Tri-Signal Structured Pruning for Large Language Models
arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.
arXiv Machine Learning
Oct 15
Your Transformer Is Secretly Elastic
The paper presents a retraining‑free technique that transforms pre‑trained transformers into elastic models capable of adapting to any computational budget on the fly. By jointly ranking every MLP neuron and attention head on a common scale and applying soft pruning across multiple sparsity targets, the method achieves smooth performance degradation up to 60% sparsity while outperforming one‑shot baselines. Experiments on 27 vision and text transformer encoders show significant parameter reductions with minimal accuracy loss and notable throughput and latency gains on GPUs, all produced in minutes.