arXiv Machine Learning September 16, 2026 By Sebastian Zhao, Minseo Kim, Coleman Hooper, Luca Manolache, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

LLM Inference in a Flash!

Read the original on arXiv Machine Learning →

The paper "LLM Inference in a Flash!" proposes an integer‑only quantization scheme and a dictionary‑based KV cache compression technique to enable large language model inference on compute‑in‑flash (CIF) devices. By eliminating floating‑point operations and reducing KV cache traffic through sparse dictionary coding, the authors achieve minimal accuracy loss while cutting dynamic KV cache traffic by 15× on Llama‑3.1‑8B and Qwen‑2.5‑7B models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 18

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

arXiv:2608. 15602v1 Announce Type: cross Abstract: While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads.

By Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, Yik-Chung Wu, Ngai Wong

llms fine-tuning efficiency

arXiv Machine Learning

Aug 20

FlashAttention for Scalable Vector Architectures

FlashAttention-V is a blocked FlashAttention implementation optimized for scalable vector architectures, designed to reduce the memory bandwidth bottleneck of transformer attention on CPUs. By fusing operations, exploiting parallelism across attention heads, and inter‑head packing, it improves vector register utilization and memory locality, enabling efficient scaling from short to very long vectors. Benchmarks on TinyLlama, Llama 3.2, Qwen2.5, and Pythia‑410M show 22×–42× speedups over scalar FlashAttention in prefill and 8×–11× in decode on a Banana Pi BPI‑F3, while also revealing quantization‑related bottlenecks that limit long‑vector scalability.

By Sonia Rani Gupta, Nikela Papadopoulou, Miquel Peric\`as

llms efficiency

arXiv AI

2d ago

Comparative Characterization of KV Cache Management Strategies for LLM Inference

arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...

By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

llms efficiency benchmarks

arXiv Computation and Language

Aug 21

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

arXiv:2608. 19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase.

By Qihang Fan, Huaibo Huang, Zhiying Wu, Bingning Wang, Ran He

llms efficiency

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

arXiv Machine Learning

1d ago

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

arXiv:2609.17573v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context set...

By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu

llms diffusion efficiency

One email a morning, machine-written

Related stories

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

FlashAttention for Scalable Vector Architectures

Comparative Characterization of KV Cache Management Strategies for LLM Inference

FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference