arXiv AI
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 9
Efficient Scaling of LLM Training with Flexible Context Parallelism
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
arXiv AI
Jul 7
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
arXiv:2509. 23722v2 Announce Type: replace-cross Abstract: Pipeline parallelism is widely used to train large language models (LLMs).
arXiv AI
Aug 11
Training Variable Long Sequences with Data-Centric Parallel
arXiv:2608. 07524v1 Announce Type: new Abstract: Training deep learning models on variable long sequences poses significant computational challenges.
arXiv AI
Jun 29
DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers
arXiv:2601. 16956v1 Announce Type: cross Abstract: The rapid growth of Large Transformer-based models, specifically Large Language Models (LLMs), now scaling to trillions of parameters, has necessitated training across thousands of GPUs using complex hybrid parallelism strategies (e.
arXiv AI
Jul 7
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.
arXiv Machine Learning
Sep 4
BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
The paper introduces BASP, a batch‑aware sequence parallelism method that partitions GPUs into disjoint groups based on micro‑batch size to reduce all‑to‑all communication. By localizing communication, BASP improves training efficiency for long‑context LLMs. Experiments on NVIDIA A100 clusters show up to 1.17‑1.31× faster end‑to‑end training on Llama and Qwen models while maintaining the same accuracy and memory usage.