arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.
By Shuowei Jin, Xueshen Liu, Jiaxin Shan, Le Xu, Tieying Zhang, Liguang Xie, Z. Morley Mao
MeshKV is a Network‑on‑Chip key‑value cache fabric designed to improve transformer decoding on tiled accelerators. It uses packetized flows, TaKV affine striping, Mare multicast with duplicate suppression, and Pad to overlap prefetch, multiply, and softmax operations, thereby converting back‑pressure into useful KV transfer. In an 8×8 FPGA prototype with LLaMA‑2‑7B and Mistral‑7B, MeshKV cuts interconnect traffic by up to 58 %, doubles KV bandwidth utilization, and boosts multi‑stream throughput by up to 1.9×.
By Dong Liu, Yanxuan Yu
The paper reports the development of 2‑bit microkernels for CPUs and mixed‑precision 2‑bit kernels for Intel Xe2 GPUs, achieving near‑roofline performance. Integrated into LLM inference pipelines, these kernels deliver up to 7× speedup over 16‑bit inference on CPUs and 6.7× on GPUs, surpassing the current state‑of‑the‑art bitnet.cpp runtime by 2.2×. The work demonstrates that ultra‑low‑bit LLM models can be deployed efficiently, offering significant gains in latency, memory, throughput, and energy consumption.
By Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke, Pradeep Dubey
arXiv:2606. 16332v1 Announce Type: cross Abstract: Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU.
By Feiyang Chen, Haibo Chen
arXiv:2609.13585v1 Announce Type: cross
Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...
By Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
arXiv:2607. 22785v1 Announce Type: cross Abstract: Apple-Silicon SoCs share CPU, GPU, and Neural Engine over one unified memory system, raising the question of whether transformer inference can be accelerated by splitting single operators across units.
By Om Mohite