arXiv:2606. 00312v1 Announce Type: cross Abstract: Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications spanning numerical analysis and modern machine learning systems.
arXiv:2606. 04405v1 Announce Type: cross Abstract: Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes.
arXiv:2606. 04028v1 Announce Type: new Abstract: The IEEE P3109 draft standard defines a parameterized family of binary floating-point formats and associated operations, with a focus on facilitating machine learning.
arXiv:2610. 01889v1 Announce Type: new Abstract: Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)?
The paper presents a second‑order theory of quantization noise for matrix multiplication, characterizing quantization formats by the variance they assign to each element. It derives a closed‑form signal‑to‑noise‑ratio law for floating‑point rounding, introduces an upper bound κ* that cannot be exceeded by any function‑preserving linear transform, and proposes KBBQ—a method that parameterizes how closely a transform approaches this bound. Experiments on W4A4 across four base models and two FP4 formats show that KBBQ outperforms the previous state of the art without extra deployment‑time computation.
arXiv:2606. 19411v3 Announce Type: replace Abstract: Selecting a fixed-size subset that maximizes the determinant of a positive semidefinite kernel is the MAP problem for a size-constrained determinantal point process and the classical maximum-entropy sampling problem.
arXiv:2505. 22988v3 Announce Type: replace-cross Abstract: The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible.
arXiv:2503. 11891v2 Announce Type: replace Abstract: We analyze the landscape and training dynamics of diagonal linear networks in a linear regression task, with the network parameters being perturbed by isotropic normal noise during training.
arXiv:2505. 01043v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved impressive performance across various domains.
arXiv:2510. 04212v4 Announce Type: replace-cross Abstract: The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models.
arXiv:2605. 11396v2 Announce Type: replace Abstract: The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization.
arXiv:2605. 13768v2 Announce Type: replace-cross Abstract: This is the second part of the work investigating quantized matrix multiplication (MatMul).
The paper revisits Kashin‑decomposition‑based weight quantization for large language models, introducing an improved algorithm that uses a sign‑randomized Discrete Cosine Transform (DCT) instead of a dense random orthogonal matrix. This change reduces per‑iteration cost from ≠(N^2) to ≠(N log N) and, combined with a greedy alternating‑update scheme, guarantees the four‑peak distribution needed for stable 2‑bit clustering while eliminating the need for multi‑restart k‑means. The resulting JAX pipeline, when paired with OPTQ‑style error compensation and QuIP‑style incoherence preprocessing, competes with state‑of‑the‑art quantization methods on OPT, Llama‑2, and Pythia at 4‑bit per channel, and remains numerically stable under stress configurations that cause other methods to diverge.