← Back to all news
arXiv Machine Learning By Qitong Wang, Xinwei Niu, Mingluo Su, Shanwei Zhao, Shiai Zhu, Huan Wang

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Read the original on arXiv Machine Learning →

SparseDecoding is a decoding‑aware pruning framework that improves large language model inference by aligning pruning with the activations seen during actual autoregressive generation. It constructs calibration matrices from dense‑model activations (excluding prefill) and introduces an optimized N:M sparse matrix‑vector kernel with bitmask indexing to accelerate decoding. Experiments on models such as Llama‑3.1‑8B, Llama‑3.3‑70B, and Qwen3‑14B/32B show consistent performance gains over standard fixed‑text calibration and up to 1.48× wall‑clock speedup on A100 GPUs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jun 9

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

arXiv:2606. 09080v1 Announce Type: new Abstract: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns.

By Haozhe Hu, Hao Wu, Anhao Zhao, Longwei Ding, Peiran Yin, Yunpu Ma, Xiaoyu Shen
llmsnlpefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 27

When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs

Sparse autoencoders (SAEs) are commonly used to interpret large language models, but their reliability after pruning is unclear. This study shows that pruning’s effect on an SAE is governed by perturbation energy, a covariance-weighted norm, and that magnitude pruning distorts the representation space by ignoring activation geometry. Activation-aware pruning methods such as Wanda and SparseGPT better preserve SAE behavior, and the authors find that middle layers are especially vulnerable, leading them to propose a layer‑wise sparsity allocation that reduces perplexity for a given sparsity level.

By Suchit Gupte, Xueru Zhang, Mohammad Mahdi Khalili
llmsefficiencysafety
More like this →