The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper identifies a missing cost term in the Tensor‑Memory Equilibrium (TME) model for emulating fp64 on GPUs: the per‑input deconstruction cost required before matrix multiplication can proceed. By adding this term and calibrating it against cuBLAS, the authors derive a closed‑form operational‑intensity threshold (OI*) below which emulation cannot match native fp64 performance. On an NVIDIA B300 GPU, this threshold is about 0.56 FLOP/B, limiting the speed‑ups of memory‑bound kernels such as GEMV, SpMV, and low‑batch GEMV to 0.3–0.9× native, and reducing the 7‑point stencil speed‑up from 3.1× to 1.8×, while dense GEMM remains unaffected.
arXiv:2608. 10103v1 Announce Type: cross Abstract: High-performance Tensor Core kernels rely on a low-level PTX pipeline built from asynchronous data movement with cp.
arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.
arXiv:2611.07496v1 Announce Type: cross Abstract: Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice...
The article argues that on AI‑optimised NVIDIA B300 GPUs and newer, the FP8 tensor‑core matrix operation—implemented via the CRT‑based Ozaki Scheme II—can become the primary substrate for matrix‑heavy FP64 kernels while maintaining FP64‑grade accuracy. It introduces the Tensor‑Memory Equilibrium (TME) model, a Roofline extension with four parameters, to show that FP8 can match native FP64 performance under certain intensity thresholds and tile‑fusion conditions. The study identifies two notable exceptions—large dense‑square DGEMM and 3‑D FFT—where additional hardware or software adjustments are required to reach the memory roof. whyItMatters":"The paper demonstrates that FP8, with appropriate reconstruction and deconstruction strategies, can replace native FP64 for high‑performance computing workloads on modern GPUs, potentially reducing hardware complexity and energy consumption while preserving accuracy."
The paper introduces GraphTok, a GPU-based tokenizer that executes the entire tokenization pipeline as a single CUDA graph, matching reference token IDs for models such as GPT‑2, Llama 3, Qwen 2.5, DeepSeek‑V3, and Gemma 3. GraphTok achieves significant speedups, encoding one million GPT‑2 tokens in 1.7–3.2 ms and outperforming the leading CPU tokenizer gigatoken by 4.7–6.0× on Llama 3. In a vLLM deployment with cached prefixes, it reduces mean time‑to‑first‑token by 2.1–2.4× and dramatically increases request throughput, sustaining 518 req/s of 121k‑token traffic versus 296 with gigatoken.