The paper introduces Layerwise Repacking, a technique that eliminates an extra data redistribution step when training hybrid‑attention models with variable‑length sequences. By keeping full‑attention layers in wide sequence‑parallel groups (SP16/32) and repacking samples into node‑local sparse children (SP1/2/4/8) within the same all‑to‑all communication, the method preserves hidden‑state continuity and avoids costly inter‑node transfers. Experiments on a 1F3S hybrid‑attention model show that repacking alone cuts end‑to‑end training time by ~6 % and boosts FLOP utilization, with an improved solver further reducing time by ~7.6 % and increasing utilization by ~4.8 %.
arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.
Kairos is a byte‑level foundation model that integrates bidirectional gated‑DeltaNet linear attention, sliding‑window attention, block‑wise Attention‑Residual aggregation, a sparse Mixture‑of‑Experts feed‑forward layer, and a multi‑scale byte‑level codec. It is trained with a uniform‑noise denoising objective over a single byte space shared by text, image, audio, video, lidar, and control streams, and the authors provide code‑derived algorithms for each component, including a cross‑session recurrent‑state memory gate. Convergence sanity checks on a 440‑byte text sentence show that the two attention variants reach a minimum loss near 10⁻³, and the implementation appears to optimize, though generalization and multimodal performance remain to be evaluated in future pretraining runs.
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv:2610.27438v1 Announce Type: cross Abstract: Sparse attention limits the keys used by each query, but finding those keys can still require a full-prefix scan at every layer. We introduce \textbf...
arXiv:2609. 18145v1 Announce Type: new Abstract: Attention pays, at every layer and for every input, the cost of searching for whom to connect.
arXiv:2608. 11231v1 Announce Type: new Abstract: LLM serving is increasingly accelerated by position-independent caching (PIC).
arXiv:2605.07191v2 Announce Type: replace-cross Abstract: A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT)...
arXiv:2606. 28560v1 Announce Type: cross Abstract: We study sparse self-attention in which each query attends to a dense local window plus a set of Fibonacci-spaced offsets, with a per-layer scalar alpha that compresses or expands the spacing.
arXiv:2608. 20210v1 Announce Type: cross Abstract: Small language models are usually built like large ones and then squeezed onto a CPU afterwards.
arXiv:2607. 15456v1 Announce Type: new Abstract: Looped, weight-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step.