← Back to all news
arXiv Machine Learning

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

The paper proposes two new 4‑bit quantization schemes for AdamW’s optimizer states, focusing on the rounding space of the second moment. Zero‑Inclusive Preconditioner‑space Stochastic Rounding (ZIP‑SR) keeps zero in the codebook and applies stochastic rounding in preconditioner space, while Zero‑Excluding EDEN (ZE‑EDEN) removes zero from the codebook and rescales the quantized block to reduce distortion. Experiments on GPT‑ and Llama‑style models from 130 M to 2.7 B parameters show that both methods close the validation‑loss gap to 32‑bit AdamW, achieving up to a 70 % reduction in the gap and outperforming TorchAO in fine‑tuning tasks.

Related stories

arXiv Machine Learning
Aug 27

Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

The paper introduces Adaptive Log‑Space (AL) quantization, a block‑wise representation that adapts the non‑zero range per block and preserves the exact‑zero invariant for non‑negative optimizer states. AL8 and AL16 are combined with signed‑momentum encodings and state‑specific precision choices, rather than a single policy for all states. Experiments on TinyLlama‑1.1B and GPT‑2 show that AL‑based quantization can match or exceed full‑precision performance while dramatically reducing optimizer‑state storage.

By Yan Wang
llmsefficiencybenchmarks
More like this →
arXiv AI
Oct 12

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.

By Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
llmsefficiency
More like this →
arXiv Machine Learning
6d ago

Noise-shaped Sampling and Annealing for Scalable On-the-edge Training of Binary and Ternary AI Models

arXiv:2610. 22621v1 Announce Type: new Abstract: The training loss of binary and ternary neural networks, expanded as a multilinear polynomial over the discrete weights, is structurally equivalent to the energy of a hierarchical high-order Ising model.

By Zihao Chen, Zhili Xiao, Rui-Jie Zhu, Yipeng Wang, Jason Eshraghian, Shantanu Chakrabartty
llmsragfine-tuningefficiencybenchmarks
More like this →
arXiv AI
Jun 19

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv:2606. 20381v1 Announce Type: new Abstract: FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements.

By Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
llmsefficiencybenchmarkssafety
More like this →