← Back to all news
arXiv Machine Learning By Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna, Daniel Silva, Rohan Ramanath

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

Read the original on arXiv Machine Learning →

The paper proposes two new 4‑bit quantization schemes for AdamW’s optimizer states, focusing on the rounding space of the second moment. Zero‑Inclusive Preconditioner‑space Stochastic Rounding (ZIP‑SR) keeps zero in the codebook and applies stochastic rounding in preconditioner space, while Zero‑Excluding EDEN (ZE‑EDEN) removes zero from the codebook and rescales the quantized block to reduce distortion. Experiments on GPT‑ and Llama‑style models from 130 M to 2.7 B parameters show that both methods close the validation‑loss gap to 32‑bit AdamW, achieving up to a 70 % reduction in the gap and outperforming TorchAO in fine‑tuning tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 27

Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

The paper introduces Adaptive Log‑Space (AL) quantization, a block‑wise representation that adapts the non‑zero range per block and preserves the exact‑zero invariant for non‑negative optimizer states. AL8 and AL16 are combined with signed‑momentum encodings and state‑specific precision choices, rather than a single policy for all states. Experiments on TinyLlama‑1.1B and GPT‑2 show that AL‑based quantization can match or exceed full‑precision performance while dramatically reducing optimizer‑state storage.

By Yan Wang
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 12

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.

By Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
llmsefficiency
More like this →
arXiv Machine Learning
6d ago

Noise-shaped Sampling and Annealing for Scalable On-the-edge Training of Binary and Ternary AI Models

arXiv:2610. 22621v1 Announce Type: new Abstract: The training loss of binary and ternary neural networks, expanded as a multilinear polynomial over the discrete weights, is structurally equivalent to the energy of a hierarchical high-order Ising model.

By Zihao Chen, Zhili Xiao, Rui-Jie Zhu, Yipeng Wang, Jason Eshraghian, Shantanu Chakrabartty
llmsragfine-tuningefficiencybenchmarks
More like this →