arXiv Machine Learning

Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference

The paper evaluates NVIDIA’s 2:4 semi‑structured sparsity against quantization for large language model inference. Experiments across multiple GPU generations and transformer GEMM operands show that when lower precision is available, adding 2:4 sparsity degrades accuracy and speed compared to simply reducing precision. However, at the lowest supported precision, 2:4 sparsity still yields a 1.3–1.7× speedup, and a retrained 2:4 model on an H100 achieves 27 % higher prefill throughput than a dense FP8 model while maintaining GSM8K accuracy.

arXiv AI

Jun 26

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang

llms diffusion efficiency multimodal

arXiv Machine Learning

Oct 3

Format-Aware Fusion for Fast FP4 Pretraining

The paper introduces format‑aware fusion, a method that co‑designs quantization producers with their scale domains and consumer layouts to fully exploit four‑bit floating‑point (FP4) Tensor Cores. Using this approach, the authors pretrain the Llama‑3‑family 8B model on 160 billion tokens, achieving up to 37.9 K tokens/s/GPU—significantly higher than standard bfloat16 or Transformer Engine FP4 baselines. The study demonstrates that FP4 performance depends on the interplay of scaling, operand packing, layout, and execution path, with downstream task rankings diverging from training‑loss rankings.

By Robert Hu

llms efficiency

arXiv AI

Jun 9

APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

arXiv:2606. 08761v1 Announce Type: cross Abstract: W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks.

By Hong Guo, Nianhui Guo, Weixing Wang, Jona Otholt, Christoph Meinel, Haojin Yang

llms efficiency benchmarks

Hugging Face Trending Papers

Jun 9

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning.

llms efficiency

arXiv AI

Oct 15

Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

arXiv:2610.14277v1 Announce Type: new Abstract: Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows a...

By Xingyu Xie, Xiaoyong Xie, Zhouchen Lin, Kim-Chuan Toh

llms efficiency benchmarks

arXiv Machine Learning

Jun 10

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

arXiv:2606. 10445v1 Announce Type: new Abstract: Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup.

By Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

llms efficiency

arXiv Machine Learning

Jun 15

Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

arXiv:2606. 14598v1 Announce Type: new Abstract: Post-training INT8 (W8A8) quantization of diffusion transformers is widely deployed as a speed optimization, yet on consumer Ampere GPUs it is frequently slower than the FP8 and NF4 alternatives it is meant to beat.

By Ali Asaria, Tony Salomone, Deep Gandhi

llms diffusion efficiency safety

arXiv Machine Learning

Oct 5

Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights

arXiv:2610.02598v1 Announce Type: new Abstract: Deploying LLMs on consumer-grade GPUs with insufficient memory to hold their weights can result in prohibitively slow inference, because decoding repea...

By JuneHyung Kim, Sankeerth Durvasula, Nandita Vijaykumar

llms efficiency

Hugging Face Trending Papers

Oct 5

StagQ: Constraint-Driven Multi-Precision Weight Quantization for LLMs

StagQ is a multi‑precision weight format for large language models that uses a 2‑bit group‑wise affine base followed by optional 1‑bit refinement planes. Each supported precision can be read as a prefix of the main stream, decoded via a shared affine map without per‑weight lookups, and a sparse side record stores the few weights that the grid handles poorly. Experiments show that StagQ outperforms baseline multi‑precision schemes on Llama‑3.1‑8B, Phi‑4, and OLMo‑2‑7B across various bit‑widths, and its GPU kernel is faster than baseline kernels for most shape‑precision combinations.

llms efficiency benchmarks

arXiv AI

Jul 17

A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi

arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.

By A. C. Opus, J. Q. Lu

rag multimodal benchmarks

arXiv Machine Learning

Jun 2

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

arXiv:2601. 22813v2 Announce Type: replace Abstract: The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs.

By Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh

llms efficiency

arXiv Machine Learning

6d ago

Memory Bandwidth, Not FLOPs: Profiling and Accelerating Query-Based Segmentation Decoders on Edge GPUs

The paper shows that query‑based segmentation decoders on edge GPUs are limited by memory bandwidth rather than FLOPs. Profiling reveals that non‑tensor‑core operations consume a large fraction of decoder time and bytes, saturating the DRAM bus. CompactInst mitigates this by using parameter‑free spatial pooling and a lightweight refinement head, achieving significant speedups on Jetson Orin Nano while maintaining accuracy.

By Oliver Busemann (Osnabr\"uck University, CLAAS E-Systems GmbH), Torben T\"oniges (CLAAS E-Systems GmbH), Martin Atzmueller (Semantic Information Systems Group, Osnabr\"uck University, German Research Center for Artificial Intelligence)

llms computer-vision

Related stories