arXiv AI

Quantization Error Is Spectrally Flat: A Single Random Probe Is a Calibrated, Data-Free Sensitivity Estimator, with Application to Budget-Targeted Mixed-Precision Quantization

arXiv Machine Learning
Oct 8

Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders

The paper introduces a label‑free mixed‑precision post‑training quantization technique for text embedders called Quantize by Drift. It uses quantization‑induced representation drift—measured by re‑encoding a corpus after quantizing a module—to estimate module sensitivity without relevance labels. Across five development embedders, drift‑based ordering of mixed‑precision plans correlates strongly with held‑out retrieval quality (macro Spearman 0.911) and outperforms uniform precision under a hard byte budget, though it does not fully replace fine‑grained redistribution strategies.

By Hyojung Han, Jongmin Kim, Seung-Hun Jeon
arXiv Machine Learning
Oct 16

Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors

The paper examines whether expert‑routing statistics from a Mixture‑of‑Experts language model can inform layer‑wise mixed‑precision quantization of routed‑expert feed‑forward networks. By building a routing‑capture pipeline for Qwen‑3.6‑35B‑A3B, the authors derive a composite sensitivity score and use coordinate descent to allocate bits per layer under a fixed budget. Experiments across flat, steep, and random allocation regimes show that routing‑guided reallocation does not outperform a uniform baseline, suggesting that routing behavior is an unreliable proxy for true quantization cost.

By Yifan Liu
Hugging Face Trending Papers
Sep 28

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

The paper investigates post‑training quantization of transformer attention blocks by optimizing a joint loss over the Q, K, V projections rather than individual weight matrices. Using this joint attention‑based objective (JAB), the authors achieve significant compression on Mistral‑7B, recovering 77‑90% of the performance gap at 3 bits, but the method fails when MLP layers are included. A role‑aware offset rule that ignores sensitivity estimates outperforms JAB on GPT‑2 and full Mistral‑7B, demonstrating that the matrix a weight belongs to is more critical than sensitivity metrics.

Hugging Face Trending Papers
5d ago

The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models

The paper investigates how various non‑quantization factors—such as engine changes, server restarts, batching differences, and kernel selection—affect the performance of quantized language models compared to their BF16 baselines. Using 2,400 public decisions from Cloudflare’s Clef‑flash (9B) and Clef (27B) models on an NVIDIA GB10, the authors quantify the noise introduced by each factor and isolate the true impact of different quantization schemes (8‑bit, FP8, Q4_K_M, NVFP4). They find that non‑quantization sources contribute up to 0.92% of decision changes, while 4‑bit quantization can alter accuracy by up to 0.95 points, and they provide code, recipes, and detailed per‑record outputs for reproducibility.