OnlineQAT introduces a two-stage framework for ultra-low-bit large language models, beginning with block-wise quantization-aware training to create a usable low-bit initialization and then applying on-policy distillation (OPD) using student-generated responses. The method employs a frozen full-precision teacher to provide a sampled reverse-KL training signal at each visited prefix, addressing the mismatch between offline recovery data and the deployed model’s self-generated prefixes. Experiments on Qwen3-1.7B show that OnlineQAT achieves the best average performance among compared quantized methods, outperforming ReasoningQAT by 2.90 and 0.44 points at W3A16 and W2A16 respectively, highlighting the benefit of student-visited states for recovery.
arXiv:2609.26708v1 Announce Type: new Abstract: Quantization-aware distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathema...
arXiv:2608.20953v1 Announce Type: cross Abstract: Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and...
arXiv:2608. 10709v1 Announce Type: new Abstract: Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation.
arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.
arXiv:2512. 18934v2 Announce Type: replace-cross Abstract: Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency.
arXiv:2606. 18114v1 Announce Type: cross Abstract: State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment.
As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model...
arXiv:2608. 06296v1 Announce Type: new Abstract: On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs).
arXiv:2609.36546v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...
arXiv:2608. 13966v1 Announce Type: new Abstract: As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality.
arXiv:2609.39223v2 Announce Type: new Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...
arXiv:2608. 02087v2 Announce Type: replace Abstract: Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration.