Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.
arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
arXiv:2511. 05313v2 Announce Type: replace Abstract: The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention.
arXiv:2603.26380v3 Announce Type: replace Abstract: The attention mechanism has been the core component in modern transformer architectures. However, the computation of standard full attention scales...
arXiv:2602.06471v2 Announce Type: replace Abstract: The architectural shape of dense Transformers has remained remarkably stable: narrow-wide-narrow feed-forward networks (FFNs) consume most non-embe...
MS-GLA introduces Multi-Scale Gated Linear Attention, distributing attention heads across multiple temporal resolutions to address representational bottlenecks in GLA Transformers. Coarser resolutions pool longer token spans for long‑range dependencies, while finer heads capture local syntax, and a learnable fusion layer recombines outputs dynamically. Experiments on language modeling, recall‑intensive tasks, and long‑context generalization show MS‑GLA outperforms GLA with up to 18.9% improvement on recall tasks and 9.5% lower perplexity on language modeling benchmarks.
arXiv:2608. 19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets.
arXiv:2606. 09862v1 Announce Type: cross Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.
arXiv:2602. 21196v2 Announce Type: replace Abstract: Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism.
arXiv:2607. 21731v1 Announce Type: new Abstract: Transformers are widely used across many domains, including natural language processing, computer vision, web search, and DNA sequence analysis.
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
UniPrefill is a prefill‑acceleration framework that works with virtually any model architecture, directly speeding up token‑level computation. It is implemented as a continuous‑batching operator and extends vLLM’s scheduling to support prefill‑decode co‑processing and tensor parallelism. The method delivers up to a 2.1× speedup in Time‑to‑First‑Token, with gains growing as concurrent requests increase.