arXiv:2608. 10010v2 Announce Type: replace Abstract: Low-precision formats usually optimize scalar fidelity while inheriting conventional product arithmetic.
arXiv:2608. 10010v1 Announce Type: new Abstract: Low-precision datatypes reduce language-model cost, but most formats optimize scalar fidelity while leaving the arithmetic induced by their products unchanged.
The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
The paper introduces a new 4‑bit floating‑point (FP4) pretraining approach that pairs E2M1 payloads with unsigned E5M3 block scales, enabling periodic tensor scaling and selective stochastic rounding while eliminating the randomized Hadamard transform. Using this method, the authors pretrained a Nemotron‑H 8B model on nearly 190 billion tokens, achieving lower training and validation losses compared to NVIDIA’s Transformer Engine. The approach also improves inference performance and demonstrates a 21.2 % increase in token throughput when certain optimizations are removed.
The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.
The paper introduces format‑aware fusion, a method that co‑designs quantization producers with their scale domains and consumer layouts to fully exploit four‑bit floating‑point (FP4) Tensor Cores. Using this approach, the authors pretrain the Llama‑3‑family 8B model on 160 billion tokens, achieving up to 37.9 K tokens/s/GPU—significantly higher than standard bfloat16 or Transformer Engine FP4 baselines. The study demonstrates that FP4 performance depends on the interplay of scaling, operand packing, layout, and execution path, with downstream task rankings diverging from training‑loss rankings.
The paper demonstrates that greedy decoding from large language models is not precision‑invariant: the same model, prompt, and decoding algorithm can produce different outputs when run in BF16 versus FP16 on identical hardware. Across six models (1.1B–7B parameters, four families, and 12B) and three benchmarks, 49–100 % of prompts diverge, with a single token flip often cascading into trajectory‑level divergence. The authors develop an empirical error‑propagation analysis that identifies the top‑two logit margin at the LM head as the key factor, and they propose a low‑overhead intervention—selective FP32 LM head recomputation—that improves exact agreement by 22–36 percentage points with less than 4 % latency overhead. "whyItMatters":"The findings reveal that precision choices can fundamentally alter model outputs, challenging the assumption of deterministic greedy decoding and highlighting the need for precision‑aware inference strategies."
arXiv:2606. 09686v1 Announce Type: cross Abstract: Numeric format proliferation in machine learning hardware -- FP8 (E4M3 and E5M2), BF16, MXFP4, microscaling block formats, and dozens of research variants -- has outpaced the availability of vendor-neutral, bit-exact reference material.
Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matrix multiplications.
arXiv:2609.26333v1 Announce Type: new Abstract: Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce me...
arXiv:2605. 10886v3 Announce Type: replace-cross Abstract: Recent GPU generations deliver significantly higher FLOPs using lower-precision arithmetic, such as FP8.
arXiv:2605. 06485v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users.