arXiv Machine Learning
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jun 3
FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
arXiv:2506. 01969v3 Announce Type: replace-cross Abstract: Efficient inference of Multi-Head Latent Attention (MLA) is challenged by deploying the DeepSeek-R1 671B model on a single Multi-GPU server.
arXiv Computation and Language
Aug 21
FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving
arXiv:2608. 19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase.
arXiv Machine Learning
Jul 27
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.
arXiv Machine Learning
Aug 20
FlashAttention for Scalable Vector Architectures
FlashAttention-V is a blocked FlashAttention implementation optimized for scalable vector architectures, designed to reduce the memory bandwidth bottleneck of transformer attention on CPUs. By fusing operations, exploiting parallelism across attention heads, and inter‑head packing, it improves vector register utilization and memory locality, enabling efficient scaling from short to very long vectors. Benchmarks on TinyLlama, Llama 3.2, Qwen2.5, and Pythia‑410M show 22×–42× speedups over scalar FlashAttention in prefill and 8×–11× in decode on a Banana Pi BPI‑F3, while also revealing quantization‑related bottlenecks that limit long‑vector scalability.
arXiv Computer Vision
Sep 18
MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration
The paper introduces MiX, a micro‑inverted‑scaling format that replaces shared exponents with shared mantissas to avoid microscaling collapse in low‑bit vision‑language models. An adaptive dual‑format inference framework (MiX‑MX) maps this format to a custom accelerator, replacing multipliers with shifters. Experiments show 4.5‑bit MiX matches or outperforms NVFP4 accuracy while improving area efficiency by 25 % and delivering 2.3–4.5× speedup with 1.4–2.9× energy savings over the Focus accelerator.
arXiv AI
Sep 17
HyQuant: Hybrid-Precision Quantization for LLM Attention
HyQuant introduces a hybrid-precision quantization framework for large language model (LLM) attention modules. It quantizes most attention states to low-bit formats while preserving a small set of vertical‑line tokens and local‑window states in full precision, guided by lightweight attention‑pattern signals. This design achieves near‑lossless accuracy across tasks while improving memory and hardware efficiency.