arXiv AI
Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 23
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.
arXiv AI
Aug 28
Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models
The paper reports the development of 2‑bit microkernels for CPUs and mixed‑precision 2‑bit kernels for Intel Xe2 GPUs, achieving near‑roofline performance. Integrated into LLM inference pipelines, these kernels deliver up to 7× speedup over 16‑bit inference on CPUs and 6.7× on GPUs, surpassing the current state‑of‑the‑art bitnet.cpp runtime by 2.2×. The work demonstrates that ultra‑low‑bit LLM models can be deployed efficiently, offering significant gains in latency, memory, throughput, and energy consumption.
arXiv AI
Jun 9
APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing
arXiv:2606. 08761v1 Announce Type: cross Abstract: W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks.
arXiv Machine Learning
3d ago
Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference
The paper evaluates NVIDIA’s 2:4 semi‑structured sparsity against quantization for large language model inference. Experiments across multiple GPU generations and transformer GEMM operands show that when lower precision is available, adding 2:4 sparsity degrades accuracy and speed compared to simply reducing precision. However, at the lowest supported precision, 2:4 sparsity still yields a 1.3–1.7× speedup, and a retrained 2:4 model on an H100 achieves 27 % higher prefill throughput than a dense FP8 model while maintaining GSM8K accuracy.
arXiv AI
Jul 17
A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi
arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.
arXiv Machine Learning
Oct 3
Format-Aware Fusion for Fast FP4 Pretraining
The paper introduces format‑aware fusion, a method that co‑designs quantization producers with their scale domains and consumer layouts to fully exploit four‑bit floating‑point (FP4) Tensor Cores. Using this approach, the authors pretrain the Llama‑3‑family 8B model on 160 billion tokens, achieving up to 37.9 K tokens/s/GPU—significantly higher than standard bfloat16 or Transformer Engine FP4 baselines. The study demonstrates that FP4 performance depends on the interplay of scaling, operand packing, layout, and execution path, with downstream task rankings diverging from training‑loss rankings.