← Back to all news
arXiv Machine Learning By Yan Wang

Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

Read the original on arXiv Machine Learning →

The paper introduces Adaptive Log‑Space (AL) quantization, a block‑wise representation that adapts the non‑zero range per block and preserves the exact‑zero invariant for non‑negative optimizer states. AL8 and AL16 are combined with signed‑momentum encodings and state‑specific precision choices, rather than a single policy for all states. Experiments on TinyLlama‑1.1B and GPT‑2 show that AL‑based quantization can match or exceed full‑precision performance while dramatically reducing optimizer‑state storage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 9

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

The paper proposes two new 4‑bit quantization schemes for AdamW’s optimizer states, focusing on the rounding space of the second moment. Zero‑Inclusive Preconditioner‑space Stochastic Rounding (ZIP‑SR) keeps zero in the codebook and applies stochastic rounding in preconditioner space, while Zero‑Excluding EDEN (ZE‑EDEN) removes zero from the codebook and rescales the quantized block to reduce distortion. Experiments on GPT‑ and Llama‑style models from 130 M to 2.7 B parameters show that both methods close the validation‑loss gap to 32‑bit AdamW, achieving up to a 70 % reduction in the gap and outperforming TorchAO in fine‑tuning tasks.

By Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves, Aman Gupta, Hiroto Udagawa, Abhishek Shivanna, Daniel Silva, Rohan Ramanath
llmsfine-tuningefficiency
More like this →
arXiv Machine Learning
Aug 11

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.

By Anthony. Lui, Mohamed. Elsaied, N. P. Savani
llmsefficiency
More like this →
arXiv AI
Jun 8

ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models

arXiv:2605. 24011v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models exhibit remarkable action generation for embodied intelligence, but their heavy compute make deployment on edge platforms impractical.

By Arash Akbari, Arman Akbari, Masih Eskandar, Qitao Tan, Yixiao Chen, Jingwu Luo, Bertha Pangaribuan, Liyun Zhang, Jennifer Dy, Geng Yuan, Xue Lin, Gaowen Liu, Stratis Ioannidis, Yanzhi Wang
agentsroboticsefficiencymultimodalbenchmarks
More like this →