The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.
The paper proposes replacing dense output projection in large language models with an HNSW-based vector index to perform maximum inner product search over token embeddings. This approach reduces memory bandwidth usage by retrieving only a small set of high-scoring tokens and can be integrated into existing decoding pipelines via sparse logits scattering. Experiments on Gemma 3, Llama 3.2, and Qwen 3 show up to 82% speed‑up in batch‑size‑one decoding while maintaining generation quality.
The paper introduces Message-Passing Quantization (MPQ), a new approach to post‑training quantization (PTQ) that treats row‑wise PTQ as posterior inference over discrete codebooks. Unlike existing solvers that commit each weight block to a single codeword early, MPQ maintains soft posterior information across blocks using vector approximate message passing and only hard‑codes decisions at the final stage. Experiments on 2‑bit lattice and 3‑/4‑bit scalar quantization for dense language models from the Llama, Qwen, and Mistral families show that MPQ improves perplexity and zero‑shot accuracy compared to strong hard‑decision baselines and works well with preprocessing techniques such as activation‑aware scaling and incoherence processing.
arXiv:2506. 20040v3 Announce Type: replace-cross Abstract: Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure.
The paper investigates how to quantize recurrent language models, which use linear recurrences instead of softmax attention. It distinguishes between extra‑state quantization (weights and activations outside the recurrent state) and intra‑state quantization (the recurrent state itself during decoding). The authors introduce LoReS, a quantization pipeline that applies value‑axis scale grouping, decay‑based mixed precision, and stochastic rounding, enabling robust 8‑bit state quantization for large models such as Qwen‑3.5, Nemotron‑H, and Kimi‑Linear, and achieving decode speedups when integrated into vLLM.
arXiv:2601. 21461v3 Announce Type: replace-cross Abstract: Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts.