arXiv Machine Learning
Jun 9
Efficient Scaling of LLM Training with Flexible Context Parallelism
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
arXiv:2606. 08476v1 Announce Type: cross Abstract: Context parallelism (CP) is essential for training large-scale, long-context language models, as it partitions sequences to reduce memory overhead.
Related stories
arXiv AI
Jun 30
Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs
arXiv:2511. 10480v3 Announce Type: replace-cross Abstract: Optimizing the performance of large language models (LLMs) on large-scale AI training and inference systems requires a scalable and expressive mechanism to model distributed workload execution.
arXiv AI
Jun 10
Piper: A Programmable Distributed Training System
arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.
arXiv Machine Learning
Jun 11
MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs
arXiv:2512. 22219v2 Announce Type: replace-cross Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel.
arXiv Machine Learning
Aug 11
ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling
arXiv:2608. 07974v1 Announce Type: new Abstract: Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy.
arXiv Machine Learning
Jul 1
HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models
arXiv:2606. 30460v2 Announce Type: replace Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.
arXiv Machine Learning
Jul 13
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
arXiv:2602. 21196v2 Announce Type: replace Abstract: Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism.
arXiv AI
Jul 10
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
arXiv:2607. 08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly.
arXiv Machine Learning
Jul 21
AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
arXiv:2607. 17913v1 Announce Type: cross Abstract: Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead.
arXiv AI
Jun 6
RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.
arXiv AI
Jun 18
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.
arXiv Machine Learning
Jun 30
HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models
arXiv:2606. 30460v1 Announce Type: new Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.