← Back to all news
arXiv AI By Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang

HyQuant: Hybrid-Precision Quantization for LLM Attention

Read the original on arXiv AI →

HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Jul 1

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du
llmsefficiencybenchmarks
More like this →
arXiv AI
Sep 30

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

arXiv:2609.38166v1 Announce Type: cross Abstract: Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Atte...

By Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica
llmsefficiency
More like this →
arXiv Machine Learning
Sep 14

Attention Quantization for Tabular Foundation Models

The paper introduces a quantization technique for tabular foundation models that focuses on converting queries, keys, and values to FP8 and employing explicit FP8 matrix multiplication to accelerate attention calculations. It emphasizes aligning quantization errors between training and test rows to avoid accuracy loss, and demonstrates up to 1.7× speedup over 16‑bit kernels with no significant accuracy degradation on TabPFN‑v3 and TabICLv2 across TabArena and BeyondArena.

By Jonas M. K\"ubler, Benjamin J\"ager, Klemens Fl\"oge, Noah Hollmann, Frank Hutter
llmsefficiency
More like this →