arXiv Machine Learning
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 23
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
arXiv:2609.26708v1 Announce Type: new Abstract: Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathema...
arXiv Machine Learning
Oct 8
OnlineQAT: On-Policy Distillation for Ultra-Low-Bit Large Language Models
OnlineQAT introduces a two-stage framework for ultra-low-bit large language models, beginning with block-wise quantization-aware training to create a usable low-bit initialization and then applying on-policy distillation (OPD) using student-generated responses. The method employs a frozen full-precision teacher to provide a sampled reverse-KL training signal at each visited prefix, addressing the mismatch between offline recovery data and the deployed model’s self-generated prefixes. Experiments on Qwen3-1.7B show that OnlineQAT achieves the best average performance among compared quantized methods, outperforming ReasoningQAT by 2.90 and 0.44 points at W3A16 and W2A16 respectively, highlighting the benefit of student-visited states for recovery.
arXiv AI
Jun 2
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
arXiv:2606. 02011v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive.
arXiv Machine Learning
Jun 2
Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
arXiv:2606. 00206v1 Announce Type: new Abstract: Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood.
Hugging Face Trending Papers
Jun 1
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Large Reasoning Models (LRMs) rely on long reasoning traces, making inference expensive. While low-bit quantization reduces per-token decoding cost, we show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count.
arXiv AI
Jul 2
When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models
arXiv:2512. 18934v2 Announce Type: replace-cross Abstract: Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency.