arXiv Machine Learning
Aug 11
Which Decisions Low-Bit Quantization Breaks, and How to Predict Them
arXiv:2608. 06564v2 Announce Type: replace Abstract: Quantization is how large language models are actually deployed, and below four bits it hurts.
Quantization Damage Is Multiplicative, Not Additive
arXiv:2608. 06564v1 Announce Type: new Abstract: Quantization is how large language models are actually deployed, and below four bits it is known to hurt.
Related stories
arXiv Machine Learning
Jul 31
Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.
arXiv Machine Learning
Jul 21
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
arXiv:2607. 16721v1 Announce Type: new Abstract: The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token.
arXiv AI
Jun 3
How Quantization Changes Interpretable Features: A Sparse Autoencoder Analysis of Language Models
arXiv:2606. 03002v1 Announce Type: cross Abstract: Quantization is a standard path to deploying large language models, and a quantized model is typically judged acceptable when its perplexity or downstream accuracy stays close to the full-precision original.
arXiv Machine Learning
Jul 15
Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate
arXiv:2607. 12266v1 Announce Type: new Abstract: Mixed-precision quantization must decide which parts of a model to keep at higher precision.
arXiv AI
Jun 8
Perplexity Can Miss SAE Feature Damage Under Quantization
arXiv:2606. 03002v2 Announce Type: replace-cross Abstract: Quantization is a standard path to deploying large language models, and quantized models are typically judged acceptable when perplexity or downstream accuracy remains close to the full-precision original.
arXiv AI
Aug 3
When Bits Break Recourse: Counterfactual-Faithful Quantization
arXiv:2605. 17160v2 Announce Type: replace-cross Abstract: Model quantization is widely used to reduce memory, latency, and deployment cost, and is typically judged by whether predictive accuracy is preserved.
arXiv AI
Jun 30
The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
arXiv:2606. 29581v1 Announce Type: cross Abstract: Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually treat these choices as fixed implementation details.
arXiv AI
Jun 10
Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation
arXiv:2606. 09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact.
Hugging Face Trending Papers
Jun 28
The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually treat these choices as fixed implementation details. This leaves a practical uncertainty: does a model that is safe at FP16 and greedy decoding remain safe after it is quantized and sampled stochastically, or do the two deployment knobs amplify one another?
Hugging Face Trending Papers
Jul 23
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked.
arXiv AI
Aug 11
Quantization Degradation in Large Language Models: A Signal-Noise Perspective
arXiv:2608. 08188v1 Announce Type: new Abstract: Post-training quantization reduces the deployment cost of large language models, yet how severely a quantized model degrades is not determined by bit-width alone.