← Back to all news
arXiv Machine Learning By Xiuyi Qin

Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference

Read the original on arXiv Machine Learning →

The paper evaluates NVIDIA’s 2:4 semi‑structured sparsity against quantization for large language model inference. Experiments across multiple GPU generations and transformer GEMM operands show that when lower precision is available, adding 2:4 sparsity degrades accuracy and speed compared to simply reducing precision. However, at the lowest supported precision, 2:4 sparsity still yields a 1.3–1.7× speedup, and a retrained 2:4 model on an H100 achieves 27 % higher prefill throughput than a dense FP8 model while maintaining GSM8K accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 26

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang
llmsdiffusionefficiencymultimodal
More like this →
arXiv Machine Learning
Oct 3

Format-Aware Fusion for Fast FP4 Pretraining

The paper introduces format‑aware fusion, a method that co‑designs quantization producers with their scale domains and consumer layouts to fully exploit four‑bit floating‑point (FP4) Tensor Cores. Using this approach, the authors pretrain the Llama‑3‑family 8B model on 160 billion tokens, achieving up to 37.9 K tokens/s/GPU—significantly higher than standard bfloat16 or Transformer Engine FP4 baselines. The study demonstrates that FP4 performance depends on the interplay of scaling, operand packing, layout, and execution path, with downstream task rankings diverging from training‑loss rankings.

By Robert Hu
llmsefficiency
More like this →