FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
By Wanqi Yang, Shiwei Liu
arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.
By Samuel Fern\'andez-Mendui\~na, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega, Salman Avestimehr
The paper introduces a quantization technique for tabular foundation models that focuses on converting queries, keys, and values to FP8 and employing explicit FP8 matrix multiplication to accelerate attention calculations. It emphasizes aligning quantization errors between training and test rows to avoid accuracy loss, and demonstrates up to 1.7× speedup over 16‑bit kernels with no significant accuracy degradation on TabPFN‑v3 and TabICLv2 across TabArena and BeyondArena.
By Jonas M. K\"ubler, Benjamin J\"ager, Klemens Fl\"oge, Noah Hollmann, Frank Hutter
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
By Beshr IslamBouli, David Jin
The paper introduces RAM‑Net, a mechanism that unifies KV‑cache quantization and linear attention by using soft assignments over a discrete address space to update continuous slot states. It shows that these soft assignments extend hard quantized matching to a separable read‑write overlap, locally approximating full‑attention similarity while enabling recurrent aggregation. The authors demonstrate that RAM‑Net can migrate pretrained Transformer weights and recover an average of 87.1% of the original accuracy across nine models using only a 500M‑token budget.
By Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing
arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.
By Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee, Murali Annavaram, Sai Praneeth Karimireddy, Sungjoo Yoo
arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.
By Yuzhen Mao, Michael Y. Li, Emily B. Fox
HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.
By Jiatong Ding, Bingxin Xing, Yu Zhang, Dian Ding, Xiaodong Yi, Xianbin Ouyang, Feihu Zhou, Kun Zhang, Zhenyu Guo, Hao Pan, Guangtao Xue, Yiming Zhang
arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.
By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du
arXiv:2609.36760v1 Announce Type: new
Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...
By Zunhai Su, Yuxuan Sun, Jianchao Tan, Tao Zhang, Ruihan Hu, Yuchen Xie, Xunliang Cai, Ngai Wong
arXiv:2608.30384v1 Announce Type: new
Abstract: By introducing RSLM (Rotated Scaled Lloyd-Max), a family of training-free vector quantization codecs compressing embeddings to 1--4 bits per dimension,...
By Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato, Jiri Isa, Igor Ginzburg
The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.
By Yi Su, Hong Liu, Guanghua Yu, Jianchen Zhu