arXiv:2605. 20982v2 Announce Type: replace-cross Abstract: AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism, and the interconnect community has responded with four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology.
By Bole Ma, Jan Eitzinger, Harald Koestler, Gerhard Wellein
The paper introduces Layerwise Repacking, a technique that eliminates an extra data redistribution step when training hybrid‑attention models with variable‑length sequences. By keeping full‑attention layers in wide sequence‑parallel groups (SP16/32) and repacking samples into node‑local sparse children (SP1/2/4/8) within the same all‑to‑all communication, the method preserves hidden‑state continuity and avoids costly inter‑node transfers. Experiments on a 1F3S hybrid‑attention model show that repacking alone cuts end‑to‑end training time by ~6 % and boosts FLOP utilization, with an improved solver further reducing time by ~7.6 % and increasing utilization by ~4.8 %.
By Jiahao Zhang, Junwei Feng, Dongliang Huang, Xiaoling Zhou, Yiqiao Liao, Chengru Song
arXiv:2608. 15299v1 Announce Type: cross Abstract: Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy.
By Lie Li, Wen Li, Junxiao Shen, Gusheng Hu
The study re‑examines a reported advantage of a routed ternary (1.58‑bit) language model over a full‑precision transformer at 60K parameters. By running controlled experiments with multiple seeds and a fixed training recipe, the authors find that the apparent benefit largely stems from the choice of baseline model shape rather than the ternary architecture itself. While the routed model does outperform other shapes at a larger 130M‑byte budget, its advantage diminishes when a plain gated diagonal‑SSM block is used, and the ternary penalty varies with architecture and quantization details.
By Gautam Veldanda
arXiv:2606. 02559v1 Announce Type: cross Abstract: Post-training compression of Large Language Models (LLMs) removes entire architectural components, either deleting them or replacing them with fitted modules.
By Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, Giovanni Iacca
arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.
By Minsik Choi, Geewook Kim