The paper introduces a label‑free mixed‑precision post‑training quantization technique for text embedders called Quantize by Drift. It uses quantization‑induced representation drift—measured by re‑encoding a corpus after quantizing a module—to estimate module sensitivity without relevance labels. Across five development embedders, drift‑based ordering of mixed‑precision plans correlates strongly with held‑out retrieval quality (macro Spearman 0.911) and outperforms uniform precision under a hard byte budget, though it does not fully replace fine‑grained redistribution strategies.
By Hyojung Han, Jongmin Kim, Seung-Hun Jeon
The paper examines whether expert‑routing statistics from a Mixture‑of‑Experts language model can inform layer‑wise mixed‑precision quantization of routed‑expert feed‑forward networks. By building a routing‑capture pipeline for Qwen‑3.6‑35B‑A3B, the authors derive a composite sensitivity score and use coordinate descent to allocate bits per layer under a fixed budget. Experiments across flat, steep, and random allocation regimes show that routing‑guided reallocation does not outperform a uniform baseline, suggesting that routing behavior is an unreliable proxy for true quantization cost.
By Yifan Liu
arXiv:2607. 23047v1 Announce Type: cross Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget.
By Ashitabh Misra, Madhav Agrawal, Arham Jain, Tarek Abdelzaher
arXiv:2607. 12266v1 Announce Type: new Abstract: Mixed-precision quantization must decide which parts of a model to keep at higher precision.
By Joshua Hill
arXiv:2608.30564v1 Announce Type: cross
Abstract: Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-ind...
By Deokjae Lee, Sihun Chu, Hyun Oh Song
arXiv:2606. 04980v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory.
By Wanqi Yang, Yuexiao Ma, Alexander Conzelmann, Xiawu Zheng, Michael W. Mahoney, T. Konstantin Rusch, Shiwei Liu
The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.
The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.
arXiv:2609.26173v1 Announce Type: new
Abstract: Many post-training quantization (PTQ) methods use layer-wise reconstruction, second-order proxy objectives, or activation-aware transformations to redu...
By Kasun Dewage, Marianna Pensky, Suranadi De Silva
arXiv:2607. 28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now.
By Fanzhe Wei, Li Liu
arXiv:2608.23816v1 Announce Type: new
Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...
By Md Romyull Islam
arXiv:2606. 00079v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory.
By Jiayu Zhao, Zihan Teng, Minhao Fan, Tianrui Ma, Wentao Ren, Song Chen, Weichen Liu