arXiv AI
Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 10
Hidden in Plain Sight: The Overlooked Significance of Canonical Elements for Extreme LLM Sparsity
arXiv:2609.06557v1 Announce Type: new Abstract: Large language models (LLMs) are often considered fragile under aggressive sparsification, and maintaining reliable performance typically requires stic...
arXiv Machine Learning
Jul 14
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
arXiv:2605. 18331v2 Announce Type: replace Abstract: Large Language Models (LLMs) have experienced significant growth and development in recent years.
arXiv Computation and Language
Aug 31
Pruning Laws for Large Language Models
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
arXiv AI
Jul 21
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
arXiv Machine Learning
Oct 15
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models
arXiv:2311.04902v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurt...
arXiv Machine Learning
Aug 27
When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
Sparse autoencoders (SAEs) are commonly used to interpret large language models, but their reliability after pruning is unclear. This study shows that pruning’s effect on an SAE is governed by perturbation energy, a covariance-weighted norm, and that magnitude pruning distorts the representation space by ignoring activation geometry. Activation-aware pruning methods such as Wanda and SparseGPT better preserve SAE behavior, and the authors find that middle layers are especially vulnerable, leading them to propose a layer‑wise sparsity allocation that reduces perplexity for a given sparsity level.