arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv:2410. 13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes.
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).
arXiv:2606. 00079v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory.
arXiv:2606. 05429v1 Announce Type: new Abstract: Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs).
arXiv:2608. 15602v1 Announce Type: cross Abstract: While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads.
arXiv:2602. 05367v3 Announce Type: replace Abstract: Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance.
arXiv:2606. 04050v1 Announce Type: cross Abstract: Existing quantization methods are fundamentally limited by rigid, integer-based bit-widths (e.
The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B. "whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."