arXiv Machine Learning August 27, 2026 By Peng Liu, Huibing Zeng, Yiqun Zhang, Yang Yi, Jigang Wu

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Read the original on arXiv Machine Learning →

The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Aug 25

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especial...

llms fine-tuning efficiency

arXiv AI

Jul 21

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.

By Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 30

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.

By Xin Yu, Cong Xie, Xunmei Liu, Tiantian Fan, Lingzhou Xue, Zhi Zhang

llms fine-tuning efficiency

arXiv AI

Jul 28

TriSP: Tri-Signal Structured Pruning for Large Language Models

arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.

By Manel Kara laoua, Soumia Bouyahiaoui, Aicha Boutorh

llms fine-tuning efficiency

arXiv AI

Jul 23

LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.

By Ashutosh Tripathi, Surya Deep Singh, Pranab Sahoo, Sriparna Saha

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 17

Stabilizing Native Low-Rank LLM Pretraining

arXiv:2602. 12429v2 Announce Type: replace Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges.

By Paul Janson, Edouard Oyallon, Eugene Belilovsky

llms rag

One email a morning, machine-written

Related stories

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

TriSP: Tri-Signal Structured Pruning for Large Language Models

LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

Stabilizing Native Low-Rank LLM Pretraining