arXiv Machine Learning July 13, 2026 By Ravi Ghadia, Maksim Abraham, Sergei Vorobyov, Max Ryabinin

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

Read the original on arXiv Machine Learning →

arXiv:2602. 21196v2 Announce Type: replace Abstract: Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 30

HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models

arXiv:2606. 30460v1 Announce Type: new Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.

By Songxin Zhang, Zejian Xie, Zhuoyang Song, Cong lin, Junyu Lu, Jiaxing Zhang, Bingyi Jing

llms fine-tuning

arXiv AI

Jun 16

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing

arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.

By Kuzey Torlak, H\"useyin Arda Arslan, An{\i}l Dervi\c{s}o\u{g}lu, Beyza Nur Deniz, Onur Boyar

llms nlp efficiency

arXiv Machine Learning

Jul 1

HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models

arXiv:2606. 30460v2 Announce Type: replace Abstract: In this paper, we aim to combine the advantages of existing sequence parallelism paradigms and overcomes their drawbacks, the most serious of which is the incapability to correctly compute causal attention on the hybrid-context packed sequences, in a stronger sequence parallelism framework.

By Songxin Zhang, Zejian Xie, Zhuoyang Song, Cong lin, Junyu Lu, Jiaxing Zhang, Bingyi Jing

llms fine-tuning

arXiv Machine Learning

Jun 9

Efficient Scaling of LLM Training with Flexible Context Parallelism

arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).

By Yifan Niu, Han Xiao, Dongyi Liu, Wei Zhou, Jia Li

llms efficiency

arXiv Computation and Language

6d ago

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification

UniPrefill is a prefill‑acceleration framework that works with virtually any model architecture, directly speeding up token‑level computation. It is implemented as a continuous‑batching operator and extends vLLM’s scheduling to support prefill‑decode co‑processing and tensor parallelism. The method delivers up to a 2.1× speedup in Time‑to‑First‑Token, with gains growing as concurrent requests increase.

By Qihang Fan, Huaibo Huang, Zhiying Wu, Bingning Wang, Ran He

llms

arXiv Computation and Language

Aug 21

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

arXiv:2608. 19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets.

By Matthias Seeger, Zeyu Zhang, Vihang Patil, Konstantinos Benidis, Sebastian Schelter

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

HSAP: A Hierachical Sequence-aware Parallelism for Hybrid-Context Generative Models

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing

HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models

Efficient Scaling of LLM Training with Flexible Context Parallelism

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention