arXiv AI
AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 19
TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
TileMix is a tile‑centric mixed‑precision attention kernel that routes score‑tile groups within fused dense attention to either FP16 or INT8 computation, using compact bitmasks to decide precision per tile. By partitioning the attention matrix into hardware‑aligned tiles and updating a shared online‑softmax state, TileMix preserves dense token connectivity without requiring training and supports grouped‑query attention, variable‑length batches, and INT8 key/value caches. Benchmarks on LLaMA, Qwen, and Vicuna show that TileMix restores long‑context quality lost with uniform INT8 and improves prefill throughput over FP16, offering a controllable accuracy‑efficiency trade‑off across model families.
arXiv Machine Learning
Aug 11
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.
arXiv AI
Jul 7
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv Machine Learning
Aug 10
CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights
arXiv:2608. 06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution.
arXiv Machine Learning
Jun 8
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv AI
Jul 28
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
arXiv:2607. 23047v1 Announce Type: cross Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget.