arXiv Machine Learning
Jun 9
Efficient Scaling of LLM Training with Flexible Context Parallelism
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
Training Variable Long Sequences with Data-Centric Parallel
arXiv:2608. 07524v1 Announce Type: new Abstract: Training deep learning models on variable long sequences poses significant computational challenges.
Related stories
arXiv Machine Learning
Aug 4
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.
arXiv AI
Jun 10
Piper: A Programmable Distributed Training System
arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.
arXiv AI
Jul 2
scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
arXiv:2506. 01883v3 Announce Type: replace-cross Abstract: Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory.
arXiv Machine Learning
Jul 24
Pipelined Gradient Coding
arXiv:2607. 20739v1 Announce Type: cross Abstract: In large-scale machine learning, distributed training commonly involves multiple workers evaluating the gradients of the model on different dataset partitions.
arXiv Machine Learning
Jul 7
ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution
arXiv:2603. 02510v2 Announce Type: replace Abstract: The transition from sequential to parallel computing is essential for modern high-performance applications but is hindered by the steep learning curve of concurrent programming.
arXiv Machine Learning
Jul 10
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
arXiv:2506. 01260v3 Announce Type: replace Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks.
arXiv AI
Jun 4
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
arXiv Machine Learning
Jul 9
Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism
arXiv:2506. 01260v2 Announce Type: replace Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks.
arXiv AI
Jun 12
Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale
arXiv:2604. 24806v2 Announce Type: replace-cross Abstract: Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws with sequence length, driving the frontier toward ultra-long User Interaction History (UIH).
arXiv Machine Learning
Jul 13
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
arXiv:2602. 21196v2 Announce Type: replace Abstract: Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism.
arXiv AI
Aug 3
Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs
arXiv:2605. 04215v3 Announce Type: replace-cross Abstract: Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead to significant throughput advantages and superior GPU utilization over the traditional autoregressive paradigm.