The paper examines why learned gates in sparse attention models offer little advantage over random gates when jointly trained with the transformer. Through experiments on a 31M-parameter transformer, the authors attribute this to routing absorption, where the model’s representations adapt to the imposed mask, diminishing the benefit of learned routing. They also explore hard masking, stochastic mask training, and the impact of trainable attention layers on gate performance, concluding that freezing the model stabilizes routing targets for effective post‑hoc sparsification.
arXiv:2607. 06601v1 Announce Type: cross Abstract: Conditional computation can decouple language model quality from per-token inference cost, yet leading techniques act on a single axis in isolation: Mixture-of-Experts (MoE) sparsifies the FFN, Mixture-of-Depths (MoD) skips whole transformer blocks, and KV-cache quantization compresses attention memory.
CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling) is a new method for long-context LLM inference that replaces costly quadratic attention prefilling with a dynamic, input-adaptive sparse routing scheme. It introduces a structural proxy, C_struct, to directly read routing decisions from the proxy attention map, eliminating the need for pooled matrix multiplication and KL divergence. Additionally, CRISP addresses the post-softmax mass cliff by using a sink-aware threshold based on the noise floor, theoretically reducing background noise accumulation to O(n). Empirical results on InfiniteBench, RULER, and LongBench show that CRISP outperforms existing sparse methods and can match or exceed exact dense attention, achieving up to a 5.30× speedup at 512k tokens and significant gains on retrieval-heavy tasks.
Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas inference saves memory only when KV entries are ph...
arXiv:2608.30320v1 Announce Type: new Abstract: We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and a...
arXiv:2608.23296v1 Announce Type: cross Abstract: Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas...
Attention-Aware Routing (AAR) augments the router in Mixture-of-Experts language models with temporal and spectral features derived from a sliding window of attention weights, thereby separating contextual information from the token’s hidden state. By keeping the base transformer frozen and training only routing parameters, AAR achieves a +3.37‑point improvement on GSM8K over a routing‑only baseline and demonstrates that routing changes propagate through the residual stream to reshape attention without directly updating the attention mechanism. The method also reduces long diverging generations, shows depth‑sensitivity affecting retrieval versus reasoning, and offers a controlled probe of routing‑relevant information across layers.
The paper introduces AMOR, an Adaptive Metacognitive Output Router that selectively activates attention in recurrent-attention hybrid models based on predictive uncertainty. By gating attention blocks with an entropy threshold derived from a running batch median and scaled standard deviation, AMOR reduces attention usage to about 40% of positions while achieving superior common-sense reasoning performance across multiple scales. The approach also improves retrieval accuracy over pure recurrent models and maintains long-context robustness, outperforming fixed-schedule hybrids under distribution shift.
The paper reports that chain‑of‑thought (CoT) supervised fine‑tuning (SFT) improves reasoning but systematically harms long‑context recall in hybrid linear‑attention models such as HypeNet and Jet‑Nemotron. Retrieval performance on the Needle‑In‑A‑Haystack benchmark drops dramatically after CoT‑SFT, especially with harder settings and longer contexts. The authors introduce QK‑Restore, a training‑free method that reinstates the query‑key projection matrices from the pre‑SFT checkpoint, which recovers long‑range recall while preserving reasoning gains.
MASKerade is a new dense‑to‑MoE training method that learns experts as sparse subnetworks of a frozen pretrained feed‑forward network (FFN) using binary masks. A token‑level router selects which masked FFNs to execute, and both the router and mask scores are jointly optimized while the underlying FFN weights remain unchanged. In experiments on five vision‑language benchmarks with Qwen and Gemma backbones, a configuration of four 2:4 experts with top‑2 routing achieved the best performance among compared baselines, demonstrating that mask learning over frozen weights is a practical alternative for constructing token‑routed MoE experts.
The paper introduces GatedV, an attention variant that applies the SiLU activation only to the value matrix before aggregation, adding no extra trainable parameters. Experiments on a decoder‑only Transformer show that, with matched parameter counts and training budgets, GatedV consistently lowers held‑out perplexity—by 2.0% at 2.33M parameters and 2.4% at 26.16M parameters—across multiple seeds. Ablations indicate that the performance gains are not fully captured by simple affine rescaling of values, and a benchmark on NVIDIA DGX Spark reports a modest 0.7% increase in full‑context forward latency for the larger model.
arXiv:2607. 07953v1 Announce Type: cross Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context.
Joint Subspace Sparse Attention (JSSA) is a new method that reduces KV‑cache traffic in long‑context LLM decoding by selecting tokens at the token level while preserving query–key structure. It learns a shared low‑dimensional query–key subspace through a tractable spectral solution and implements a gather‑free indexed‑attention kernel that supports per‑KV‑head selections. Across various attention architectures, JSSA maintains near full‑KV accuracy, avoids generation‑length inflation, and can achieve up to 6.43× decoding throughput on a single GPU for 128K context, even at 700B‑parameter scale.