OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
Read the original on arXiv Machine Learning →OnlineQAT introduces a two-stage framework for ultra-low-bit large language models, beginning with block-wise quantization-aware training to create a usable low-bit initialization and then applying on-policy distillation (OPD) using student-generated responses. The method employs a frozen full-precision teacher to provide a sampled reverse-KL training signal at each visited prefix, addressing the mismatch between offline recovery data and the deployed model’s self-generated prefixes. Experiments on Qwen3-1.7B show that OnlineQAT achieves the best average performance among compared quantized methods, outperforming ReasoningQAT by 2.90 and 0.44 points at W3A16 and W2A16 respectively, highlighting the benefit of student-visited states for recovery.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.