arXiv:2606. 04920v1 Announce Type: new Abstract: Quantizing deep neural networks is essential for efficient inference on resource-constrained devices.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
Squeeze10-LLM is a staged mixed‑precision post‑training quantization framework that reduces 16‑bit LLM weights to an average of 1.6 bits per weight by assigning 80% of weights to 1 bit and 20% to 4 bits. It introduces Post‑Binarization Activation Robustness (PBAR), a weight significance metric that considers activation impact, and Full Information Activation Supervision (FIAS), a strategy that preserves activation information to limit error propagation. Experiments on LLaMA and LLaMA2 demonstrate that Squeeze10‑LLM achieves state‑of‑the‑art performance for sub‑2‑bit weight‑only quantization, raising average accuracy from 43% to 56% on six zero‑shot classification tasks.
The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
The paper introduces the Quantization Analysis Tool, a system built on the ONNX framework that streamlines quantization workflows for deep learning models. It offers layer‑wise sensitivity analysis, visualizations of weight and activation distributions, and guidance for selecting precision levels to balance model size, latency, and accuracy. Experiments on various neural network architectures show that the tool improves quantized accuracy and overall deployment efficiency.
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.