The paper reports a post‑training ternarization of the 4‑billion‑parameter Qwen model, achieving an effective 1.641‑bit representation for 81.62 % of its weights while keeping activations at 16‑bit precision. Accuracy drops from 64.5 % to 54.7 % across ten capability tests, with uneven degradation (e.g., BoolQ 84.6 % of teacher performance, ARC‑Challenge 43.8 %). After packing the ternary planes, the model size shrinks from 8.29 GiB to 3.96 GiB with negligible change in perplexity, though inference speed is not improved.
By Anirudh Malik, M Sparsh Mehra, Poojith Devan
arXiv:2609.33923v2 Announce Type: replace-cross
Abstract: A single random Gaussian probe gives an unbiased estimate of the squared Frobenius norm of a layer's quantization error. The estimator is wel...
By I Kennedy, T Kennedy
arXiv:2607. 01478v1 Announce Type: cross Abstract: We measured quantization-induced decision-boundary changes using local logit-margin radii, first-order boundary displacement, normal variation, slice-boundary Jaccard distance, grid prediction changes, multiclass junction counts, and low-margin boundary-band flips.
By O. M. Kiselev
arXiv:2606. 02823v1 Announce Type: new Abstract: Two-bit weight quantization is attractive for memory-efficient LLM inference, but the standard W2 level set {-2,-1,0,+1} often collapses under aggressive W2A4/KV4 settings.
By Chi-Wei Huang, Chia-Chi Tsai
ShamAN-Q is a sub‑1‑bit post‑training quantization technique that builds on NanoQuant by replacing its diagonal reconstruction geometry with a dense curvature metric inspired by the Shampoo optimizer. For each linear weight, it fits a Kronecker product to the empirical Fisher information matrix of a small calibration set via Kullback–Leibler minimization, yielding a Mahalanobis reconstruction loss. The method updates continuous ADMM steps to Sylvester equations while keeping the discrete projection and deployment format unchanged, and it redistributes uniform rank across layers, achieving lower perplexity on Qwen3‑Base at roughly 1 bpw and matching or improving zero‑shot accuracy on the Eleuther LM Evaluation Harness.
By Jonathan Mei, Sang Hyub Kim, Oliver Knitter, Chi Chen, Martin Roetteler
arXiv:2606. 08635v1 Announce Type: new Abstract: Prefill-decode (PD) disaggregation decouples prompt processing from token generation, but it also turns the key-value (KV) cache into a network payload.
By Yang Pengju
The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.
The paper introduces an error‑diffusion (ED) optimizer that replaces the full‑precision latent copy of binary‑weight language models with a small 2‑8‑bit residual buffer, reducing training state per weight from 97 bits to 3‑9 bits—up to 19× less. ED accumulates unrealized weight updates in the buffer and flips the binary weight when the accumulated value crosses a threshold aligned with the quantization grid, enabling efficient training on large transformers with minimal loss compared to latent‑weight baselines. The method scales to models with up to 19 M parameters on enwik8 and 70 M on wikitext‑103, maintaining performance within a few percent of reference while allowing on‑device adaptation within a 275 MB memory budget.
By Pierre-Olivier Bonin
The paper reports a large‑scale post‑training ternarisation of the Qwen3 language model, extending a conversion pipeline from the 4B to the 8B variant. Using KOTMS rotation, E2M‑ATQ adaptive ternarisation, and GPTQ‑style error compensation, the authors achieve a 1.361× perplexity ratio across three corpora and retain 78.5% of the FP16 accuracy on zero‑shot tasks, with the 8B model outperforming the 4B by 8.9 percentage points. The study also demonstrates lossless lattice‑aware packing, producing an 8.24 GiB checkpoint that preserves perplexity, and shows that direct packed execution can reach 15.52 tokens/s in 7.35 GiB, though packed GEMV remains slower than FP16 cuBLAS.
By Anirudh Malik, M Sparsh Mehra, Poojith Devan
arXiv:2607. 10137v1 Announce Type: new Abstract: Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision.
By Prateek Singh
Scale-QLoRA introduces a method for merging LoRA adapters into native 4‑bit microscaling language models without altering the quantized code plane. By training only the per‑block scale field and freezing the E2M1 code, the merged model remains code‑invariant and can be deployed as a single checkpoint. Experiments on four models and tasks show that Scale‑QLoRA achieves accuracy‑lossless merging comparable to merge‑aware QAT‑LoRA, while offering benefits such as exact rollback, code‑plane deduplication, and a 125× faster scale‑only task swap.
By Tung-Ling Li, Jiale Huang, Lee-Chi Wang, Janaki Ram Gotei
The paper presents a 4‑bit quantization recipe, Minima: NVFP4 W4A4, that fully quantizes all linear layers—including the Gated DeltaNet (GDN) recurrent blocks—of the 27‑billion‑parameter Qwen3.8 LLM. Across a suite of benchmarks (perplexity, MMLU‑Pro, GSM8K, AIME'25, GPQA‑Diamond, LiveCodeBench, and RULER retrieval), the quantized model matches BF16 performance within seed noise while being 17.5 GiB in size and 14–19 % faster at prefill. The authors attribute this success to four mechanisms: block‑scaling of residuals, robust gate projections, the delta‑rule recurrence’s noise‑plateau behavior, and the per‑token quantization cost’s dilution over long contexts.
By Sergii Kozyrev, Davyd Maiboroda