The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.
By Mohammed Nayeem Khan, Allan Kevin T
The paper proposes a dynamic weight‑precision strategy for local decoding, storing 8‑bit weights but using only the top 4 bits during most steps and escalating to full precision when the low‑precision output distribution is uncertain (high entropy or small top‑two margin). Experiments on Qwen3‑0.6B with the GSM8K benchmark show that a 15% escalation budget captures 78% of low‑precision argmax flips, outperforming a random gate, and that an entropy gate at 16% escalation achieves 62.7% accuracy with only 0.87 of the weight‑byte cost of using all‑high precision. The study demonstrates that selective precision escalation can reduce computational cost while maintaining competitive performance.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2608.23816v1 Announce Type: new
Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...
By Md Romyull Islam
arXiv:2610. 01889v1 Announce Type: new Abstract: Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)?
By Yohan Chatelain (Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada), Pablo de Oliveira Castro (Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France)
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.
By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
By Hamish Ogilvy