arXiv Machine Learning
ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 12
Hand-Written PTX Tensor-Core GEMM Kernels: A Multi-Precision Study on NVIDIA L4
arXiv:2608. 10103v1 Announce Type: cross Abstract: High-performance Tensor Core kernels rely on a low-level PTX pipeline built from asynchronous data movement with cp.
arXiv AI
Jun 12
MiniMax Sparse Attention
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv AI
Jul 17
A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi
arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.
arXiv Computation and Language
2d ago
FlyWheel: Attention Kernels Built for the TPU
FlyWheel introduces optimized TPU kernels for softmax and linear attention, reducing vector work and improving matrix operand reuse. By overlapping independent computations, FlyWheel achieves up to 4.14× speedups over the RPA kernel and 1.80× over Splash Attention, reaching 81% of peak BF16 throughput on TPU v6e. It also boosts Gated DeltaNet prefill by up to 3.61× and enhances vLLM serving of Qwen3.8-27B prefill throughput by up to 1.56× while preserving accuracy.
arXiv AI
Oct 19
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
arXiv:2610. 23796v1 Announce Type: new Abstract: Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads.
arXiv Machine Learning
Sep 22
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.