AsyncFlow is an asynchronous streaming reinforcement learning framework designed to improve the post‑training phase of large language models. It introduces a distributed data storage and transfer module that enables panoramic data management and fine‑grained scheduling, allowing automated pipeline overlapping and dynamic load balancing. The framework also employs an asynchronous producer‑consumer workflow to reduce computational idleness by deferring parameter updates within staleness thresholds, and it is architecturally decoupled from training and inference engines, providing modular, customizable user interfaces. Experiments show an average throughput improvement of 1.59× over the state‑of‑the‑art baseline.
By Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, Jianping Wu
Maia 200 is a software‑defined dataflow system that delivers high performance AI acceleration, achieving 10,145 Tflop/s in FP4 and 5,072 Tflop/s in FP8 within a 750 W TDP and 7 TB/s HBM bandwidth. It exemplifies a new class of Software Defined Locally Accessed Dataflow Architectures (SDLA), which program dataflow engines to orchestrate specialized memories and data‑movement engines, shifting focus from thread‑centric to data‑movement‑centric design. The system offers significant cost and energy savings while supporting massive parallelism for AI inference workloads, positioning it as a compelling solution for next‑generation high‑performance computing.
By Sherry Xu, Marco Heddes, Jackson Peng, Tom Savell, Monica Tang, Prashant Ranjan, Jesse Benson, Ofer Dekel, Saurabh Dighe, Anupama Kurpad, Artour Levin, Matthew Mattina, George Petre, Cheng Tang, Yuan Yu, Li Zhang, Torsten Hoefler
arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.
By Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral
arXiv:2608. 13076v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands.
By Divya Jyoti Bajpai, Kishan Kumar Upadhyay, Manjesh Kumar Hanawal
arXiv:2402.03694v3 Announce Type: replace-cross
Abstract: Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryp...
By Shinan Liu, Ted Shaowang, Gerry Wan, Jeewon Chae, Jonatas Marques, Sanjay Krishnan, Nick Feamster
arXiv:2607. 22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes.
By Xin Yang, Yemin Wang, Mingda Liu, Letian Li, Shuaishuai Cao, Zhengxiao He, Ryan Dong
arXiv:2601. 20408v2 Announce Type: replace-cross Abstract: Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset.
By Nicholas Santavas, Kareem Eissa, Patrycja Cieplicka, Piotr Florek, Matteo Nulli, Stefan Vasilev, Seyyed Hadi Hashemi, Antonios Gasteratos, Shahram Khadivi
arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.
By Gangmuk Lim, Wanyu Zhao, Brighten Godfrey, Jiaxin Shan, Le Xu, Liguang Xie
arXiv:2609.08307v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...
By Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos
arXiv:2509. 23722v2 Announce Type: replace-cross Abstract: Pipeline parallelism is widely used to train large language models (LLMs).
By Jihu Guo, Tenghui Ma, Wei Gao, Peng Sun, Xun Chen, Jiaxing Li, Zhisheng Ye, Yuyang Jin, Dahua Lin
The paper introduces ETH, a training system that implements Heterogeneous Pipeline Parallelism (HPP) to achieve fine‑grained data balance across heterogeneous pipelines. ETH uses a hierarchical workload balance framework and block‑wise gradient aggregation to reduce communication overhead, while also providing optimizations that cut configuration search time. Experiments show ETH can double training speed and cut configuration generation overhead by up to 56× compared to existing systems.
By Mianjie Yu, Zhongmin Zhao, Zizhao Mo, Jianxiong Liao, Wenxuan Li, Yulin Qiao, Huanle Xu, Wenquan Yang, Yongqiang Yang, Zeren Li, Ruifeng Tang, Chengzhong Xu
TIDE (Temporal Incremental Draft Engine) is a serving‑engine‑native framework that integrates online draft adaptation into high‑performance LLM inference. By reusing intermediate hidden states from the target model as training signals, TIDE avoids extra target model computation and serving‑time overhead, activating speculation and draft training only when beneficial. On heterogeneous GPU clusters, TIDE achieves up to 1.66× higher throughput than no‑speculation baselines, reduces training time by up to 3.02×, cuts storage needs by 24×, and improves system throughput by up to 1.22×.
By Jiyoung Park, Hankyu Jang, Changseok Song, Wookeun Jung