Q-PACE is a dynamic precision allocation method for quantization-aware training that uses a second-order sensitivity model to predict loss increases from quantization noise. By periodically recalculating per-layer curvature coefficients and reassigning precision, it improves mixed-precision training for large language models up to 4 B parameters, achieving comparable loss with lower memory usage. The approach shows that quantization sensitivity is predictable by depth and layer type, enabling infrequent, inexpensive recalibration.
By Alexandra Volkova, Matin Ansaripour, Erik Schultheis, Christoph H. Lampert, Dan Alistarh
arXiv:2602. 03120v2 Announce Type: replace-cross Abstract: Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune.
By Yinggan Xu, Kajetan Schweighofer, Risto Miikkulainen, Xin Qiu
arXiv:2602. 10431v4 Announce Type: replace Abstract: Large language models (LLMs) demand substantial computational and memory resources, posing challenges for efficient deployment.
By Kanghyun Noh, Jinheon Choi, Yulhwa Kim
arXiv:2606. 09734v1 Announce Type: cross Abstract: Training parameterised quantum circuits (PQCs) on quantum hardware is bottlenecked by the measurement cost of gradient estimation, which under the parameter-shift rule scales linearly in the number of trainable parameters and dominates the total shot budget of training at scale.
By Brian Coyle, Snehal Raj, Virag Umathe, El Amine Cherrat, Elham Kashefi
arXiv:2608. 06916v1 Announce Type: new Abstract: Quantized Neural Networks~(QNN) with low-bitwidth data have proven promising in efficient storage and computation on edge devices.
By Zijun Jiang, Yangdi Lyu
arXiv:2605. 25054v2 Announce Type: replace-cross Abstract: Deploying deep neural networks on resource-constrained 6G edge devices demands aggressive compression with minimal accuracy loss.
By Ayush K. Varshney, Konstantinos Vandikas, \v{S}ar\=unas Girdzijauskas, Adam Orucu, Aneta Vulgarakis Feljan
arXiv:2608. 01997v1 Announce Type: new Abstract: Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on.
By Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri
The paper presents a target‑centric survey of Quantization‑Aware Training (QAT), a technique that simulates quantization during model training to produce low‑bit models with accuracy comparable to full‑precision ones. It systematically reviews existing QAT methods using a target‑centric taxonomy, highlighting differences in error characteristics, numerical formats, and strategy transferability across targets. The survey also summarizes QAT evaluation paradigms, discusses optimization and deployment challenges, and outlines potential future research directions.
By Jiamin Song, Mengjie Zhao, Zijing Wang, Yongkang Liu, Qian Li, Shi Feng, Feiliang Ren, Daling Wang, Hinrich Sch\"utze
arXiv:2607. 20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.
By Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, Tijmen Blankevoort
arXiv:2607. 18101v1 Announce Type: new Abstract: On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks.
By Mateusz Piechocki, Alessandro Capotondi, Marek Kraft
arXiv:2405. 04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge.
By Yijiang Pang, Shuyang Yu, Bao Hoang, Jiayu Zhou
ShiftQAT is a quantization‑aware training workflow designed for the MAX7800x family of fixed‑function CNN accelerators. It enforces the accelerator’s numerical contract—power‑of‑two, per‑tensor symmetric scales, 8‑bit activations and biases, integer‑only arithmetic, and a single statically configured shift per layer—by training weights and biases on a joint power‑of‑two grid and resolving dependencies so that activation scales propagate statically. Compared to other methods, ShiftQAT uniquely satisfies all contract constraints while matching the accuracy of less constrained QAT approaches, achieving ImageNet performance within 2.9 percentage points of FP32 at W4A8 and enabling fast, energy‑efficient inference on MAX78002 silicon.
By Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem Ulkar