MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries

MoSAR introduces a mixture of semantic attention regimes that learns an adaptive, distance‑dependent attention geometry from data, rather than predefining sparse or local patterns. The model uses input‑conditioned routers to select short, medium, or global regimes, creating a continuous attention field that can be discretized for efficient inference. Experiments show that MoSAR achieves lower‑reach attention without sacrificing language‑modeling quality, improving perplexity over dense RoPE and outperforming baselines like ALiBi, while remaining stable under top‑1 discretization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• efficiency