arXiv AI
SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 5
Tailoring the Quantization Space for 1-Bit KV Cache Compression
arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
arXiv Machine Learning
1d ago
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...
arXiv Machine Learning
Jun 8
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv AI
Jul 7
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv AI
Jun 6
Channel-Wise Mixed-Precision Quantization for Large Language Models
arXiv:2410. 13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes.
arXiv Machine Learning
Jun 16
MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).