TR-PTQ is a post‑training quantization method that enables high‑accuracy, integer‑only inference for transformer models. It identifies learned scale parameters in LayerNorm and compounded approximations in GELU as the main sources of accuracy loss, while SoftMax is robust to aggressive quantization. By reformulating these operations with shared Taylor Region exponential and logarithm primitives and a calibration‑free, outlier‑aware optimization for LayerNorm, TR‑PTQ eliminates the need for floating‑point units for nonlinearities and achieves less than 1.5% absolute accuracy degradation on vision and language benchmarks.
arXiv:2505. 01043v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved impressive performance across various domains.
arXiv:2506. 04985v2 Announce Type: replace Abstract: Large language models (LLMs) require substantial compute, and thus energy, at inference time.
arXiv:2606. 07819v1 Announce Type: new Abstract: Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications.
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
arXiv:2505. 22988v3 Announce Type: replace-cross Abstract: The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible.
arXiv:2509. 10334v2 Announce Type: replace-cross Abstract: Vision Transformers (ViTs) have recently achieved strong results in semantic segmentation, yet their deployment on resource-constrained devices remains limited due to their high memory footprint and computational cost.
The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
arXiv:2606. 10890v1 Announce Type: cross Abstract: Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations.
arXiv:2610.16368v1 Announce Type: cross Abstract: Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over...
The paper investigates how the choice of reconstruction loss, specifically mean squared error (MSE), affects optimization in sequential post‑training quantization (PTQ) of large language models (LLMs). It identifies an Optimization Imbalance where MSE causes reconstruction loss magnitudes—and thus gradient magnitudes—to vary dramatically across quantization stages, leading to uneven parameter updates. The authors propose that root mean squared error (RMSE) variants, which implicitly normalize gradients, can decouple optimization strength from loss scale and outperform MSE as a drop‑in replacement.
arXiv:2601. 21626v2 Announce Type: replace-cross Abstract: Post Training Quantization (PTQ), a mainstream model compression technique, often leads to the paradoxical 'low error, high loss' phenomenon because it focuses solely on minimizing quantization error.