The paper investigates whether disagreement between two quantized versions of the same language model can serve as an error signal for tasks such as abstention and escalation. Using Qwen3-0.6B and Qwen3-1.7B, the authors compare an 8‑bit copy with lower‑precision copies (3–6 bits) on multiple-choice datasets, measuring error detection via AUROC, risk‑coverage, and ECE. Results show modest gains over entropy alone, with additional forward passes offering limited improvement, and the study provides a deployment recipe and discusses scenarios where the method is ineffective.
By Mohammed Nayeem Khan, Allan Kevin T
The paper proposes a dynamic weight‑precision strategy for local decoding, storing 8‑bit weights but using only the top 4 bits during most steps and escalating to full precision when the low‑precision output distribution is uncertain (high entropy or small top‑two margin). Experiments on Qwen3‑0.6B with the GSM8K benchmark show that a 15% escalation budget captures 78% of low‑precision argmax flips, outperforming a random gate, and that an entropy gate at 16% escalation achieves 62.7% accuracy with only 0.87 of the weight‑byte cost of using all‑high precision. The study demonstrates that selective precision escalation can reduce computational cost while maintaining competitive performance.
By Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri
arXiv:2608.23816v1 Announce Type: new
Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...
By Md Romyull Islam
arXiv:2610. 01889v1 Announce Type: new Abstract: Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)?
By Yohan Chatelain (Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada), Pablo de Oliveira Castro (Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France)
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.
By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
By Hamish Ogilvy
arXiv:2607. 04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16.
By Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou
arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.
By Zekun Wu, Swati Dhiman, Adriano Koshiyama
arXiv:2608.28911v1 Announce Type: new
Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length....
By Daeha Lee, Do-Hyung Kim, Jae-Hong Kim
The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.
By Tatiana Petrova, Andrei Mazniak, Radu State
The paper identifies a hidden mismatch in reinforcement‑learning fine‑tuning of large language models: when training uses fp32 master weights but rollouts are served from a bf16 copy produced by round‑to‑nearest conversion, most per‑step updates are too small to affect the served weights, causing a lag of 50–150 steps. The authors propose Fair Cast, a one‑line change that exports the master weights with an unbiased rounding rule (stochastic rounding or error feedback) and computes the loss on the served copy, eliminating the lag without adding hyperparameters and incurring only a modest computational cost. Across multiple models and optimizers, Fair Cast consistently outperforms round‑to‑nearest, improves checkpoint consistency, and reveals the implicit regularization effect of the lag.
By Andrew Gao, William Gao
The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.