arXiv:2608. 05253v1 Announce Type: new Abstract: Quantized orthogonal fine-tuning (qoft) enables parameter-efficient adaptation of low-bit language models by learning structured activation rotations before frozen quantized weights.
arXiv:2602. 03120v2 Announce Type: replace-cross Abstract: Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune.
arXiv:2606. 09927v1 Announce Type: cross Abstract: Post-training quantization (PTQ) is one of the most practical ways to reduce the serving cost of Large Language Models (LLMs), but activation quantization remains difficult because outlier-dominated channels lead to large quantization errors.
REAL-Q introduces a new post‑training quantization approach for large language models that replaces the traditional single closed‑form second‑order solver with a fine‑grained, dynamic block‑wise gradient descent applied after every 128‑column block. By aligning the surrogate loss with the end‑to‑end objective and using a sliding window for smooth cross‑layer transitions, REAL‑Q mitigates error propagation and information misalignment. Experiments on LLaMA‑3.1 and Qwen3 show up to ~49% reduction in end‑to‑end KL divergence compared to state‑of‑the‑art methods.
arXiv:2607. 04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16.
arXiv:2606. 10531v1 Announce Type: cross Abstract: Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs).
arXiv:2605.11222v2 Announce Type: replace Abstract: Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (...
HARP (Hadamard‑Preconditioned Adaptive Rotation Processor) is a learnable, structured two‑sided orthogonal processor that replaces fixed randomized Hadamard transforms in post‑training quantization of large language models. By representing rotations as sparse butterfly‑like block‑orthogonal stages and supporting mixed‑radix schedules, HARP adapts the quantization basis to each layer and calibration distribution while maintaining full‑precision equivalence. Across 2–4‑bit settings on Llama models from 1B to 70B, HARP consistently improves perplexity, delivers the strongest zero‑shot gains at 2 bits, and preserves deployment efficiency—achieving 128 tokens per second on Llama 2 7B at 2 bits, roughly 90% of RHT throughput and over twice the speed of FP16.
The paper introduces a post‑training softmax reparameterization technique that selects a functionally equivalent output head before quantization. By subtracting a scalar multiple of the vocabulary‑row mean from each output row and tuning this coefficient via validation KL, the method preserves the full‑precision softmax distribution while enabling efficient W4 quantization. Experiments on seven heads show significant error reductions and latency improvements, with the approach remaining complementary to other quantization strategies and transferable across datasets.
arXiv:2607. 10137v1 Announce Type: new Abstract: Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision.
arXiv:2602. 05367v3 Announce Type: replace Abstract: Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance.
arXiv:2609.00224v1 Announce Type: cross Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existi...
The paper presents a target‑centric survey of Quantization‑Aware Training (QAT), a technique that simulates quantization during model training to produce low‑bit models with accuracy comparable to full‑precision ones. It systematically reviews existing QAT methods using a target‑centric taxonomy, highlighting differences in error characteristics, numerical formats, and strategy transferability across targets. The survey also summarizes QAT evaluation paradigms, discusses optimization and deployment challenges, and outlines potential future research directions.