arXiv Machine Learning September 4, 2026 By Bigyan Ghimire, Jon C. Calhoun

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

Read the original on arXiv Machine Learning →

The paper introduces BASP, a batch‑aware sequence parallelism method that partitions GPUs into disjoint groups based on micro‑batch size to reduce all‑to‑all communication. By localizing communication, BASP improves training efficiency for long‑context LLMs. Experiments on NVIDIA A100 clusters show up to 1.17‑1.31× faster end‑to‑end training on Llama and Qwen models while maintaining the same accuracy and memory usage.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jun 9

Efficient Scaling of LLM Training with Flexible Context Parallelism

arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).

By Yifan Niu, Han Xiao, Dongyi Liu, Wei Zhou, Jia Li

llms efficiency

arXiv AI

Aug 11

Training Variable Long Sequences with Data-Centric Parallel

arXiv:2608. 07524v1 Announce Type: new Abstract: Training deep learning models on variable long sequences poses significant computational challenges.

By Geng Zhang, Xuanlei Zhao, Kai Wang, Yang You

arXiv Computation and Language

Sep 23

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing laten...

By Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia, Yutaka Watanobe, Fang Liu

llms rag efficiency benchmarks

arXiv Machine Learning

Jul 1

HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models

arXiv:2606. 30460v2 Announce Type: replace Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.

By Songxin Zhang, Zejian Xie, Zhuoyang Song, Cong lin, Junyu Lu, Jiaxing Zhang, Bingyi Jing

llms fine-tuning

arXiv AI

Jun 2

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.

By Zifan He, Rui Ma, Yizhou Sun, Jason Cong

llms rag

arXiv Machine Learning

Jun 30

HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models

arXiv:2606. 30460v1 Announce Type: new Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.

By Songxin Zhang, Zejian Xie, Zhuoyang Song, Cong lin, Junyu Lu, Jiaxing Zhang, Bingyi Jing

llms fine-tuning

One email a morning, machine-written

Related stories

Efficient Scaling of LLM Training with Flexible Context Parallelism

Training Variable Long Sequences with Data-Centric Parallel

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models