Hugging Face Trending Papers
The Undetected Damage of Quantization on Retrieval and How to Fix It
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Sep 22
The Undetected Damage of Quantization on Retrieval and How to Fix It
arXiv:2609.24322v1 Announce Type: new Abstract: We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate r...
arXiv Machine Learning
Sep 2
The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.
arXiv Machine Learning
Jun 2
Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
arXiv:2606. 00206v1 Announce Type: new Abstract: Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood.
arXiv AI
Jul 10
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
arXiv:2607. 08734v1 Announce Type: new Abstract: Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity.
arXiv Machine Learning
Sep 11
Why Does Post-Training Quantization Work?
Post‑training quantization compresses large language models by storing weights at reduced precision, introducing errors into hidden states that could accumulate with depth. However, pretrained models accumulate far less hidden‑state error than randomly initialized ones, largely preserving downstream performance. The study identifies two key mechanisms: (1) each layer’s new error tends to oppose inherited error, partially canceling it, and (2) the LM‑head geometry preserves high‑rank token scores, mitigating output changes.
Hugging Face Trending Papers
Jul 9
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization.