arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap.
HARTS (Hybrid‑Attention RL over Tree Structures) is a new system that jointly plans microbatches, data‑parallel replica assignments, and microbatch‑slot schedules to efficiently train agentic reinforcement learning models with hybrid attention over arbitrary rollout trees. It uses prefix compression and a linear‑time algorithm for chunkwise linear attention to avoid recomputing shared prefixes, enabling activation recomputation and bounded state replay while preserving trajectory‑wise training benefits. In experiments on an Agentic RL workload derived from SWE‑bench tasks, HARTS delivers 4.81–4.87× speedups in forward, backward, and gradient computations across multiple parallel configurations, with numerical differences comparable to baseline self‑rerun variation and a similar reward trend over the first 120 training steps.
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
BF1 is a deterministic block‑aligned dyadic sparse‑attention retrofit designed to reduce the cost of causal attention in long‑context transformers. It combines a small exact local neighborhood, a global first block, and logarithmically spaced historical blocks, achieving O(n log n) token interactions per layer with O(log n) communication depth. On an NVIDIA RTX PRO 6000 Blackwell GPU, BF1 outperforms dense attention for 2K–4K tokens and delivers up to a 10.91× prefill speedup at 32K tokens, while retrofitting eight of 28 Qwen3‑0.6B layers reduces first‑token latency by up to 15.3% at 32K tokens and yields the lowest perplexity among compared sparse and dense training protocols.
arXiv:2607. 14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment.
arXiv:2608. 19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase.
FlashBoB introduces an I/O‑efficient algorithm for exact backward‑over‑backward (BoB) in softmax attention, enabling precise second‑order differentiation without large intermediate tensors. By exploiting a hierarchical affine structure, the method confines computation to on‑chip tiles and limits off‑chip memory traffic, achieving θ(N² d²/M) HBM usage. Experiments show FlashBoB scales to sequence lengths of 262K on a single A100 GPU, outperforming prior exact baselines and FlashBack by up to 6.3×.
arXiv:2608. 14498v1 Announce Type: new Abstract: Vision-language models (VLMs) enable embodied agents to reason and act from visual observations and language instructions.
arXiv:2605. 15250v3 Announce Type: replace-cross Abstract: Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly.
arXiv:2606. 27732v1 Announce Type: cross Abstract: Discrete diffusion language models (dLLMs) recover masked tokens in parallel, offering significant speedups over autoregressive (AR) generation.
arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
arXiv:2603. 17484v2 Announce Type: replace-cross Abstract: Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval.