The paper introduces Aether‑7B‑5Attn, a 6.59B‑parameter mixture‑of‑experts Transformer that arranges seven distinct sequence‑mixing mechanisms in a 7×7 Latin square across 49 layers, ensuring each mechanism appears exactly once per row and column. Experiments with a 4×4 Latin‑square proxy and larger 1.514B‑parameter models show that balanced heterogeneous composition across depth yields near‑optimal performance, while clustering mechanisms or using a homogeneous stack incurs measurable penalties. The authors provide detailed cost profiles, multilingual evaluations, a causal‑safety audit, and release all code and weights.
arXiv:2605. 20982v2 Announce Type: replace-cross Abstract: AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism, and the interconnect community has responded with four families of mitigations: predictive sample placement, adaptive expert relayout, hierarchical collectives, and EP-aware topology.
The paper introduces Layerwise Repacking, a technique that eliminates an extra data redistribution step when training hybrid‑attention models with variable‑length sequences. By keeping full‑attention layers in wide sequence‑parallel groups (SP16/32) and repacking samples into node‑local sparse children (SP1/2/4/8) within the same all‑to‑all communication, the method preserves hidden‑state continuity and avoids costly inter‑node transfers. Experiments on a 1F3S hybrid‑attention model show that repacking alone cuts end‑to‑end training time by ~6 % and boosts FLOP utilization, with an improved solver further reducing time by ~7.6 % and increasing utilization by ~4.8 %.
arXiv:2608. 15299v1 Announce Type: cross Abstract: Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy.
The study re‑examines a reported advantage of a routed ternary (1.58‑bit) language model over a full‑precision transformer at 60K parameters. By running controlled experiments with multiple seeds and a fixed training recipe, the authors find that the apparent benefit largely stems from the choice of baseline model shape rather than the ternary architecture itself. While the routed model does outperform other shapes at a larger 130M‑byte budget, its advantage diminishes when a plain gated diagonal‑SSM block is used, and the ternary penalty varies with architecture and quantization details.
arXiv:2606. 02559v1 Announce Type: cross Abstract: Post-training compression of Large Language Models (LLMs) removes entire architectural components, either deleting them or replacing them with fitted modules.
arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.
arXiv:2607. 18323v1 Announce Type: cross Abstract: Exhaustive site-by-site interventions on a neural network's computational graph -- activation-patching sweeps, circuit-discovery searches, systematic ablation studies -- mutate the graph at every candidate site, and their cost is dominated by recomputation after each mutation.
arXiv:2605. 23393v2 Announce Type: replace-cross Abstract: Mechanistic interpretability of transformers requires identifying not just which components matter but how they compose into the computational route that produced a prediction.
The paper proposes MixerLoop, a looped language model that repeats only the Gated DeltaNet mixer while applying the dense feed‑forward network (FFN) once, instead of repeating the entire layer stack. Using Iterative Transport Rank (ITR) to measure cross‑position influence, the authors show that MixerLoop retains most of the performance gains of full recurrence while reducing FLOPs by 45.9%. Experiments on 15M and 110M parameter models demonstrate that MixerLoop outperforms full‑block recurrence on aggregate CORE metrics.
The paper investigates how the four‑stream manifold‑constrained hyper‑connection (mHC) residual pathway in DeepSeek‑V4‑Flash is actually used. It finds that read/write routing is concentrated, typically involving only two streams per block, and that the dominant stream shifts across layers while representations stay directionally distinct. Residual mixing is modest, mainly in early layers, and late mixing contributes little to performance, whereas early mixing is crucial for perplexity and task scores.
arXiv:2610.09372v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts....
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.