← Back to all news
arXiv Machine Learning By Shangzhen Zhu, Muyan Hu, Tomasz Kozlowski

Pretraining Transformers with Quantized Softmax in Attention

Read the original on arXiv Machine Learning →

The paper investigates how quantizing the softmax operation in Transformer attention affects pretraining. It explores K‑interval attention, varying grid calibration, interpolation versus hard rounding, and surrogate placement relative to normalization, deriving backward rules for each case. Experiments on a 124M‑parameter model show that choices like fixed‑window calibration with a post‑normalization surrogate or a pre‑normalization surrogate can reduce validation loss gaps compared to standard softmax.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 11

Attn-QAT: 4-Bit Attention With Quantization-Aware Training

arXiv:2603. 00040v3 Announce Type: replace-cross Abstract: Achieving reliable 4-bit attention is a prerequisite for end-to-end FP4 computation on emerging FP4-capable GPUs, yet attention remains the main obstacle due to FP4's tiny dynamic range and attention's heavy-tailed activations.

By Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, Hao Zhang
llmsdiffusionefficiency
More like this →
arXiv Machine Learning
Jun 16

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.

By Zhongzhu Zhou, Qingyang Wu, Junxiong Wang, Mayank Mishra, Shuaiwen Leon Song, Ben Athiwaratkun, Chenfeng Xu
llmsefficiencybenchmarkssafety
More like this →
arXiv AI
Sep 28

Softmax Reparameterization for Output-Head Quantization

The paper introduces a post‑training softmax reparameterization technique that selects a functionally equivalent output head before quantization. By subtracting a scalar multiple of the vocabulary‑row mean from each output row and tuning this coefficient via validation KL, the method preserves the full‑precision softmax distribution while enabling efficient W4 quantization. Experiments on seven heads show significant error reductions and latency improvements, with the approach remaining complementary to other quantization strategies and transferable across datasets.

By Asim Kadav, Christian Flores, Chirag Arora, Varun Kotte, Hongbo Zheng, Lan Yan, Priya Shanmugasundaram, Tracy Holloway King
llmsefficiency
More like this →
arXiv Machine Learning
Sep 21

Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

The paper investigates how gating the value pathway in attention mechanisms provides two missing capabilities of softmax attention: abstention and noise filtering. Experiments on models ranging from 10M to 350M parameters show that abstention benefits smaller models while noise filtering becomes more advantageous as models scale, and that combining both primitives yields the best performance across all sizes. The authors also demonstrate that the gates effectively suppress interference and that each gate type has a distinct blind spot, all while adding negligible parameters and preserving compatibility with key‑value caching.

By Richard Zhe Wang
llms
More like this →