TileQuant is a training‑free post‑training quantization framework that assigns precision to 2‑D weight tiles aligned with hardware. It uses offline calibration to score tiles based on activation statistics and loss sensitivity, keeping the highest‑scoring tiles at 16‑bit while quantizing the rest to 8‑bit. The resulting precision map enables efficient integer and 16‑bit kernels that reduce memory traffic and achieve 2.66× the original model’s generation throughput while preserving or surpassing quality across multiple language‑modeling tasks.
arXiv:2610.07457v1 Announce Type: cross Abstract: Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GP...
TileMix is a tile‑centric mixed‑precision attention kernel that routes score‑tile groups within fused dense attention to either FP16 or INT8 computation, using compact bitmasks to decide precision per tile. By partitioning the attention matrix into hardware‑aligned tiles and updating a shared online‑softmax state, TileMix preserves dense token connectivity without requiring training and supports grouped‑query attention, variable‑length batches, and INT8 key/value caches. Benchmarks on LLaMA, Qwen, and Vicuna show that TileMix restores long‑context quality lost with uniform INT8 and improves prefill throughput over FP16, offering a controllable accuracy‑efficiency trade‑off across model families.
MoBiQuant introduces a Mixture-of-Bits quantization framework that allows large language models to be inferred at varying precisions during runtime. It addresses the poor generalization of existing post‑training quantization methods by mitigating precision‑dependent outlier migration through a recursive residual quantization scheme and a token‑aware router that selects the optimal precision per token. Experiments demonstrate that MoBiQuant matches or exceeds single‑precision PTQ performance while providing significant memory savings and up to 1.34× throughput gains over current any‑precision approaches.
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv:2607. 00908v1 Announce Type: new Abstract: Mixed-precision quantization (MPQ) has become a key technique for deploying large language models under stringent memory and compute constraints.
arXiv:2605. 26660v2 Announce Type: replace Abstract: Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging.
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).
arXiv:2507. 23035v4 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, but demand substantial memory and compute resources during inference.
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
arXiv:2611.07256v1 Announce Type: cross Abstract: Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interac...
ProQuant introduces a progressive quantization‑aware training framework for low‑bit multi‑modal large language models (MLLMs), addressing the performance drop seen with post‑training quantization at low bit‑widths. The method includes a self‑relabeling scheme to align fine‑tuning data with pre‑training distributions and a progressive block activation mechanism that prioritizes multi‑modal understanding, guided by a vision‑sensitive factor to identify critical layers. Experiments on 2B–8B models show ProQuant outperforms existing techniques, achieving up to a 6.6% accuracy gain in W4A8 Qwen3‑VL‑2B‑Instruct with only ~1% drop from full precision.
arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.