The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
By Jundong Hu, Shekar Ramachandran
arXiv:2608. 06564v1 Announce Type: new Abstract: Quantization is how large language models are actually deployed, and below four bits it is known to hurt.
By Zekun Wu, Swati Dhiman, Adriano Koshiyama
arXiv:2607. 12266v1 Announce Type: new Abstract: Mixed-precision quantization must decide which parts of a model to keep at higher precision.
By Joshua Hill
arXiv:2607. 16721v1 Announce Type: new Abstract: The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token.
By Anik Jha
arXiv:2606. 19558v1 Announce Type: new Abstract: Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality.
By Milo\v{s} Nikoli\'c, Ali Hadi Zadeh, Enrique Torres Sanchez, Andreas Moshovos
arXiv:2606. 09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact.
By Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou
arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.
By Zekun Wu, Swati Dhiman, Adriano Koshiyama
The paper investigates how activation quantisation affects different computational aspects of language models. While perplexity remains a reliable overall metric, it fails to reveal that induction accuracy stays high while retrieval accuracy drops significantly. Experiments show that the damage depends on more than just error magnitude, and that rotating the basis can recover induction performance even at low bit‑rates.
By Anish Sathyanarayanan
arXiv:2609.24322v1 Announce Type: new
Abstract: We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate r...
By Luca Zhou, Alessandro Zirilli, Daniele Solombrino, Roberto Dess\`i, Emanuele Rodol\`a
The study investigates how post‑training quantization (PTQ) affects proactive interference (PI) in large language models. Using bitsandbytes, the authors compare FP16, INT8, and INT4/NF4 precision across three instruction‑tuned models and find that INT4 quantization markedly degrades accuracy under high interference, with INT8 also incurring a smaller penalty in two of the three models. The degradation is linked to increased same‑key intrusion errors and originates in the quantized transformer backbone rather than the output layer.
By Shayan Shahrabi-Farahani (Shahid Beheshti University, Tehran, Iran), Dara Rahmati (Shahid Beheshti University, Tehran, Iran)
arXiv:2605. 17160v2 Announce Type: replace-cross Abstract: Model quantization is widely used to reduce memory, latency, and deployment cost, and is typically judged by whether predictive accuracy is preserved.
By Chaymae Yahyati, Ismail Lamaakal, Khalid El Makkaoui, Ibrahim Ouahbi
arXiv:2607. 28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now.
By Fanzhe Wei, Li Liu