arXiv Machine Learning

Perplexity Cost Understates What Activation Quantisation Breaks

The paper investigates how activation quantisation affects different computational aspects of language models. While perplexity remains a reliable overall metric, it fails to reveal that induction accuracy stays high while retrieval accuracy drops significantly. Experiments show that the damage depends on more than just error magnitude, and that rotating the basis can recover induction performance even at low bit‑rates.

arXiv Machine Learning

Aug 20

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

The study investigates how post‑training quantization (PTQ) affects proactive interference (PI) in large language models. Using bitsandbytes, the authors compare FP16, INT8, and INT4/NF4 precision across three instruction‑tuned models and find that INT4 quantization markedly degrades accuracy under high interference, with INT8 also incurring a smaller penalty in two of the three models. The degradation is linked to increased same‑key intrusion errors and originates in the quantized transformer backbone rather than the output layer.

By Shayan Shahrabi-Farahani (Shahid Beheshti University, Tehran, Iran), Dara Rahmati (Shahid Beheshti University, Tehran, Iran)

llms nlp efficiency benchmarks

arXiv AI

Jun 10

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

arXiv:2606. 09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact.

By Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou

llms efficiency benchmarks safety

arXiv AI

Jun 8

Perplexity Can Miss SAE Feature Damage Under Quantization

arXiv:2606. 03002v2 Announce Type: replace-cross Abstract: Quantization is a standard path to deploying large language models, and quantized models are typically judged acceptable when perplexity or downstream accuracy remains close to the full-precision original.

By Evan Duan

llms efficiency safety

arXiv Machine Learning

Sep 2

The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally

The paper investigates where post‑training quantization (PTQ) harms large language models (LLMs) and how to best allocate a limited precision budget. By causally raising each layer to 8‑bit precision across nine open‑weight models, the authors find that quantization damage is diffuse rather than concentrated in specific task circuits or weight statistics, and that globally refining quantization granularity outperforms selectively protecting the most recoverable layers. They also observe that the residual accuracy loss is budget‑limited and that peak recovery locations correlate with architecture within families but not across families.

By Jundong Hu, Shekar Ramachandran

llms efficiency

arXiv Machine Learning

Jul 31

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

arXiv:2607. 27275v1 Announce Type: new Abstract: Post-training quantization to 4-bit weights is widely reported to be nearly lossless.

By Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

llms agents efficiency benchmarks safety

arXiv Machine Learning

Aug 24

Nothing Changed but the Model: CellFill -- Bounded In-Cell Learning for Bit-Identical, Revocable Updates to Quantized LLMs

arXiv:2608.20873v1 Announce Type: new Abstract: Every way of teaching a deployed language model something new -- full fine-tuning, adapter merging, model editing -- replaces the released checkpoint,...

By Zifeng Liu, Zhiyong Du, Yaxin Lu, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing

llms fine-tuning efficiency

arXiv Machine Learning

Jul 7

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models

arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.

By Hamish Ogilvy

llms efficiency

arXiv Computation and Language

6d ago

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

arXiv:2609.30820v1 Announce Type: cross Abstract: Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes...

By Nux Li

llms efficiency

arXiv Machine Learning

Sep 23

Component Type, Not Reconstruction Error, Predicts Attention Quantization Sensitivity

arXiv:2609.26173v1 Announce Type: new Abstract: Many post-training quantization (PTQ) methods use layer-wise reconstruction, second-order proxy objectives, or activation-aware transformations to redu...

By Kasun Dewage, Marianna Pensky, Suranadi De Silva

llms efficiency

arXiv AI

Sep 2

QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization

arXiv:2609.00224v1 Announce Type: cross Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existi...

By Yipin Guo, Arun M George, Jie Fu, Tareq Mahmoud, Sixue Xing, Siddharth Joshi

llms efficiency

arXiv Machine Learning

Jul 15

Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate

arXiv:2607. 12266v1 Announce Type: new Abstract: Mixed-precision quantization must decide which parts of a model to keep at higher precision.

By Joshua Hill

efficiency

arXiv Machine Learning

Sep 2

Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance

The paper evaluates the effectiveness of tolerance‑based conformance tests for INT8 quantized GEMM kernels used in large language models. By injecting nine faults into a Qwen3‑1.7B reference pipeline, the authors show that most faults shift outputs by at most one bfloat16 spacing, rendering a tolerance of one spacing blind to these errors. They further demonstrate that requantizing weight scales to the nearest power of two aligns CUTLASS and Triton implementations bit‑for‑bit and produces identical token sequences, with only minor perplexity changes.

By Teng-Ruei Chen

llms efficiency

Related stories