DanLing NestedTensor is a PyTorch tensor abstraction that embeds multi‑ragged structure directly into the tensor, allowing packed values to carry partition information and logical dimension order. This design enables broadcasting, feature transformations, and reductions to automatically respect ragged axes while preserving the same representation through autograd and both eager and compiled execution. Benchmarks on an A100 show significant speedups—up to 3.39× over padding for BERT models and 2.40–4.32× for a Pairformer‑style workload—while dramatically reducing peak memory usage.
By Zhiyuan Chen
arXiv:2608. 00029v1 Announce Type: cross Abstract: The performance of deep learning models at scale relies heavily on how effectively high-level mathematical operations are mapped to underlying physical hardware.
By Adwaid Suresh, Aparna A, Harshini V M, Jona Delcy C A, Killi Uma Maheswara Rao, Ram Charan Golla, Surendra Vendra
arXiv:2610.07742v1 Announce Type: cross
Abstract: Optimized kernels such as FlashAttention and FlashDecoding are crucial for accelerating today's large models. Most of them are handwritten by experts...
By David Pissarra, Jinkun Lin, Haitian Jiang, Aurojit Panda, Jinyang Li
arXiv:2610.15124v1 Announce Type: cross
Abstract: Foundation models need to support diverse inference budgets, yet training a separate model for each budget duplicates pretraining effort. Elastic mod...
By Can Jin, Yan Kang, Yifan Gong, Chongjian Ge, Mingze Xu, Yuanjun Xiong, Dimitris N. Metaxas, Hongwu Peng
The paper investigates how to optimally partition streaming inference models into megakernels and conventional graph components. It introduces a framework that predicts the best megakernel composition without full end‑to‑end runs, showing that multiple megakernels or a fully conventional graph often outperform a single megakernel across various models and batch sizes. The study defines four model categories based on megakernel stability and demonstrates significant latency and throughput gains on an H100 NVL GPU using a three‑megakernel design.
By Aditya Chatterjee, Anand Joseph X M, Sanjay Singh Chauhan
The paper introduces a looped causal encoder-decoder (CED) architecture that increases model depth without enlarging the key-value (KV) cache across layers and rounds. By placing recurrence in the encoder and allowing sparse encoder layers to generate global KV, the design achieves language‑modeling gains while maintaining long‑context scaling. Experiments show that the Looped CED matches the memory footprint of an unlooped CED and recovers most of a looped Transformer’s performance with significantly less KV, enabling higher throughput on large‑token prompts.
By Jiaqi Leng, Minshen Zhang, Zhaofeng Sun, Yu-Yang Qian, Peng Zhao, Hao Zhang