The paper introduces GatedV, an attention variant that applies the SiLU activation only to the value matrix before aggregation, adding no extra trainable parameters. Experiments on a decoder‑only Transformer show that, with matched parameter counts and training budgets, GatedV consistently lowers held‑out perplexity—by 2.0% at 2.33M parameters and 2.4% at 26.16M parameters—across multiple seeds. Ablations indicate that the performance gains are not fully captured by simple affine rescaling of values, and a benchmark on NVIDIA DGX Spark reports a modest 0.7% increase in full‑context forward latency for the larger model.
arXiv:2606. 21848v2 Announce Type: replace-cross Abstract: Transformer architectures form the foundation of modern natural language processing, making it crucial to address the efficiency and scalability limitations of the standard QKV attention mechanism.
MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.
arXiv:2610.07940v1 Announce Type: cross Abstract: Looped language models apply the same stack of layers T times to each token, which deepens the model without adding parameters but multiplies its key...
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.
arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
The paper presents a retraining‑free technique that transforms pre‑trained transformers into elastic models capable of adapting to any computational budget on the fly. By jointly ranking every MLP neuron and attention head on a common scale and applying soft pruning across multiple sparsity targets, the method achieves smooth performance degradation up to 60% sparsity while outperforming one‑shot baselines. Experiments on 27 vision and text transformer encoders show significant parameter reductions with minimal accuracy loss and notable throughput and latency gains on GPUs, all produced in minutes.
Multi-State Linear Attention (MSLA) improves long‑sequence modeling by compressing context into separate memory states, but its aggregation only uses mean alignment, ignoring distribution. MERLIN (MomEnt Routing for multi‑state LINear attention) enhances this by tracking first and second moments of each segment’s keys, enabling queries to recover both mean and variance of alignment for more content‑aware aggregation. Experiments on Gated DeltaNet‑1.3B and Mamba‑2‑780M show MERLIN consistently outperforms existing aggregation methods on long‑context tasks, achieving significant gains on RULER and LongBench while using less memory.
Multi-State Linear Attention (MSLA) improves long‑sequence modeling by compressing context into separate memory states, but its aggregation only uses mean alignment, ignoring distribution within segments. MERLIN (MomEnt Routing for multi‑state LINear attention) enhances this by tracking first and second moments of each segment’s keys, enabling queries to recover both mean and variance of alignment for more content‑aware aggregation. Experiments on Gated DeltaNet‑1.3B and Mamba‑2‑780M demonstrate that MERLIN consistently outperforms existing aggregation methods on long‑context tasks, achieving significant accuracy gains while using less memory.
arXiv:2608. 19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets.
arXiv:2606. 04032v1 Announce Type: cross Abstract: Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role.
arXiv:2610.04753v2 Announce Type: replace-cross Abstract: Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Spa...