Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers

Orthros introduces a phase‑aware heterogeneous attention mechanism that dynamically switches between linear‑complexity prefilling and full‑attention decoding, reducing the asymptotic cost compared to standard Transformers in long‑context scenarios. The method supports efficient Transformer‑to‑Orthros conversion, achieving higher performance than the original model and delivering a 21.9× prefilling speedup while maintaining strong in‑context recall, including 128K‑scale needle‑in‑a‑haystack tasks. These results demonstrate Orthros’s potential as a more efficient and flexible alternative for industrial Transformer applications.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms