The paper identifies a missing cost term in the Tensor‑Memory Equilibrium (TME) model for emulating fp64 on GPUs: the per‑input deconstruction cost required before matrix multiplication can proceed. By adding this term and calibrating it against cuBLAS, the authors derive a closed‑form operational‑intensity threshold (OI*) below which emulation cannot match native fp64 performance. On an NVIDIA B300 GPU, this threshold is about 0.56 FLOP/B, limiting the speed‑ups of memory‑bound kernels such as GEMV, SpMV, and low‑batch GEMV to 0.3–0.9× native, and reducing the 7‑point stencil speed‑up from 3.1× to 1.8×, while dense GEMM remains unaffected.
By Harun Bayraktar, John Gunnels, Peter Caday
arXiv:2608. 10103v1 Announce Type: cross Abstract: High-performance Tensor Core kernels rely on a low-level PTX pipeline built from asynchronous data movement with cp.
By Matt J. Borowski, Blazej Osinski
arXiv:2607. 14568v1 Announce Type: cross Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.
By A. C. Opus, J. Q. Lu
arXiv:2611.07496v1 Announce Type: cross
Abstract: Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice...
By Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli, Stephen Ebert
The article argues that on AI‑optimised NVIDIA B300 GPUs and newer, the FP8 tensor‑core matrix operation—implemented via the CRT‑based Ozaki Scheme II—can become the primary substrate for matrix‑heavy FP64 kernels while maintaining FP64‑grade accuracy. It introduces the Tensor‑Memory Equilibrium (TME) model, a Roofline extension with four parameters, to show that FP8 can match native FP64 performance under certain intensity thresholds and tile‑fusion conditions. The study identifies two notable exceptions—large dense‑square DGEMM and 3‑D FFT—where additional hardware or software adjustments are required to reach the memory roof.
whyItMatters":"The paper demonstrates that FP8, with appropriate reconstruction and deconstruction strategies, can replace native FP64 for high‑performance computing workloads on modern GPUs, potentially reducing hardware complexity and energy consumption while preserving accuracy."
By Satoshi Matsuoka
The paper introduces GraphTok, a GPU-based tokenizer that executes the entire tokenization pipeline as a single CUDA graph, matching reference token IDs for models such as GPT‑2, Llama 3, Qwen 2.5, DeepSeek‑V3, and Gemma 3. GraphTok achieves significant speedups, encoding one million GPT‑2 tokens in 1.7–3.2 ms and outperforming the leading CPU tokenizer gigatoken by 4.7–6.0× on Llama 3. In a vLLM deployment with cached prefixes, it reduces mean time‑to‑first‑token by 2.1–2.4× and dramatically increases request throughput, sustaining 518 req/s of 121k‑token traffic versus 296 with gigatoken.
By Mohammad Siavashi, Pascal Schwinte, Ali Banaei, Gerald Q. Maguire Jr., Marco Chiesa, Dejan Kosti\'c
The paper shows that query‑based segmentation decoders on edge GPUs are limited by memory bandwidth rather than FLOPs. Profiling reveals that non‑tensor‑core operations consume a large fraction of decoder time and bytes, saturating the DRAM bus. CompactInst mitigates this by using parameter‑free spatial pooling and a lightweight refinement head, achieving significant speedups on Jetson Orin Nano while maintaining accuracy.
By Oliver Busemann (Osnabr\"uck University, CLAAS E-Systems GmbH), Torben T\"oniges (CLAAS E-Systems GmbH), Martin Atzmueller (Semantic Information Systems Group, Osnabr\"uck University, German Research Center for Artificial Intelligence)
The paper introduces format‑aware fusion, a method that co‑designs quantization producers with their scale domains and consumer layouts to fully exploit four‑bit floating‑point (FP4) Tensor Cores. Using this approach, the authors pretrain the Llama‑3‑family 8B model on 160 billion tokens, achieving up to 37.9 K tokens/s/GPU—significantly higher than standard bfloat16 or Transformer Engine FP4 baselines. The study demonstrates that FP4 performance depends on the interplay of scaling, operand packing, layout, and execution path, with downstream task rankings diverging from training‑loss rankings.
By Robert Hu
arXiv:2611.08363v1 Announce Type: new
Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision dat...
By Jake Hyun, Ziyu Zhao, Jae-sun Seo, Mohamed S. Abdelfattah
arXiv:2607. 22785v1 Announce Type: cross Abstract: Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units.
By Om Mohite
arXiv:2606. 06510v1 Announce Type: cross Abstract: Conventional HPC dogma holds that native hardware FP64 silicon is the irreducible foundation of scientific computing -- the "holy grail" of double-precision simulation.
By Satoshi Matsuoka
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
By Jie Li, Chenxin Jia, Jinliang Shen, Cunzhuang Liu, Ruiyi Ding, Jianwen Xian, Kang He, Chengru Song