← Back to all news
arXiv AI By Yanshuo Chen, Chenxi Liu, Reza Shirkavand, Heng Huang

MPQ: A Message-Passing View of Post-Training Quantization

Read the original on arXiv AI →

The paper introduces Message-Passing Quantization (MPQ), a new approach to post‑training quantization (PTQ) that treats row‑wise PTQ as posterior inference over discrete codebooks. Unlike existing solvers that commit each weight block to a single codeword early, MPQ maintains soft posterior information across blocks using vector approximate message passing and only hard‑codes decisions at the final stage. Experiments on 2‑bit lattice and 3‑/4‑bit scalar quantization for dense language models from the Llama, Qwen, and Mistral families show that MPQ improves perplexity and zero‑shot accuracy compared to strong hard‑decision baselines and works well with preprocessing techniques such as activation‑aware scaling and incoherence processing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
1d ago

Trainable Vector Quantization for Large Language Models

The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.

By Anastasiia Prutianova, Alina Bogdanova, Mikhail Shutov, Dmitrii Poimanov, Anatoly Gorkov, Ilia Zharikov
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Oct 22

Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories

The paper introduces OnQ, a low‑bit recovery pipeline for quantized large language models that uses on‑policy distillation and candidate‑wise Dynamic KL to address trajectory divergence and distributional misalignment. OnQ starts with block‑wise calibration, then lets the quantized student generate responses and receive full‑precision teacher supervision on its own prefixes, dynamically weighting forward and reverse KL losses per candidate. Experiments on Qwen3 models show that OnQ outperforms baselines on nine benchmarks, especially long‑horizon reasoning, and produces shorter, less truncated responses.

By Boyuan Zheng, Jing Xiong, Chenwei Wu, Hui Shen, Zesen Zhao, Hei Ting Una Chan, Xueshen Liu, Zhen Zhang, Liyue Shen, Zechun Liu, Z. Morley Mao
llmsefficiencybenchmarks
More like this →