← Back to all news
arXiv AI By Vincent-Daniel Yun, Woosang Lim

Forward-Free LLM Depth Pruning via Weight Redundancy

Read the original on arXiv AI →

The paper introduces Weight-Redundancy Pruning (WRP), a forward‑free depth‑pruning technique for large language models that estimates inter‑layer redundancy using only checkpoint weights. WRP compares attention outputs and MLP down‑projection weights across layers, combining pairwise similarities with relative projection‑scale information to guide layer grouping and block selection. Experiments show that WRP consistently outperforms existing forward‑free magnitude pruning methods and approaches the performance of activation‑based pruning across various pruning settings, model families, and downstream tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computer Vision
Aug 27

SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

SHIFT-LLM is a training‑free post‑pruning correction framework that inserts a Linear Residual Adapter (LRA) at each depth‑pruned site in large language models. Each LRA preserves the original residual identity while adding a lightweight affine correction calibrated via closed‑form least‑squares regression on a small held‑out set, thereby approximating the hidden state that would have been produced by the removed block. Experiments across multiple model families and benchmarks show that SHIFT‑LLM consistently recovers accuracy lost to depth pruning, achieving gains up to +15.7 points on Llama‑3.1‑8B‑Instruct with only a few hundred calibration samples and no gradient computation.

By Ali Bahri, Hang Li, Hongliang Li, Zhitang Chen
llmsfine-tuningefficiencybenchmarks
More like this →
arXiv AI
Jun 9

Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs

arXiv:2605. 15491v2 Announce Type: replace-cross Abstract: Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation.

By Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee
llmsefficiencysafety
More like this →
arXiv Machine Learning
Aug 10

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

arXiv:2608. 06901v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pre-training, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments.

By Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung
llmsefficiencymultimodal
More like this →