arXiv Machine Learning
BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 16
MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).
arXiv AI
Sep 2
QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
arXiv:2609.00224v1 Announce Type: cross Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existi...
arXiv Computation and Language
4d ago
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
MoBiQuant introduces a Mixture-of-Bits quantization framework that allows large language models to be inferred at varying precisions during runtime. It addresses the poor generalization of existing post‑training quantization methods by mitigating precision‑dependent outlier migration through a recursive residual quantization scheme and a token‑aware router that selects the optimal precision per token. Experiments demonstrate that MoBiQuant matches or exceeds single‑precision PTQ performance while providing significant memory savings and up to 1.34× throughput gains over current any‑precision approaches.
arXiv AI
Jun 9
APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing
arXiv:2606. 08761v1 Announce Type: cross Abstract: W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks.
arXiv AI
Jul 7
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv AI
Jun 26
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.