arXiv Computation and Language August 25, 2026 By Joshua Nunley

TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal Language Modeling

Read the original on arXiv Computation and Language →

The paper introduces TANGO, a new Transformer variant that replaces the standard self‑attention and feed‑forward sublayers with a single cross‑token gated residual update. Each token generates a SwiGLU gate, and query‑key similarities weight these gates to rescale destination features, yielding a quadratic‑time model. A windowed variant, WANGO, limits the gated interactions to a recent window and uses prefix statistics for older tokens, achieving linear‑time complexity while maintaining competitive performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

llms

arXiv Machine Learning

1d ago

TANGO: Treating Tokens as Operators

The paper introduces TANGO, a Token‑Aggregated Nonlinear Gating Operator that blends cross‑token mixing and token‑wise transformation in transformer architectures. By computing a nonlinear gate per source token and averaging these gates for each destination, TANGO forms a source‑conditioned linear operator that improves predictive performance. Experiments on web text, formal mathematics, and code show that full‑prefix TANGO achieves the lowest test negative log‑likelihood across 16 settings, while a narrower variant offers substantial throughput gains with only a modest increase in loss.

By Joshua Nunley

llms efficiency

arXiv AI

Jun 16

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing

arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.

By Kuzey Torlak, H\"useyin Arda Arslan, An{\i}l Dervi\c{s}o\u{g}lu, Beyza Nur Deniz, Onur Boyar

llms nlp efficiency

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

arXiv AI

Jul 21

Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

arXiv:2607. 17419v1 Announce Type: cross Abstract: Linear attention promises constant-time recurrent inference but degrades sharply on associative recall.

By Ayoub Ghriss, Sourav Chakraborty

llms rag efficiency benchmarks

arXiv Machine Learning

Jun 3

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

arXiv:2602. 03681v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios.

By Difan Deng, Andreas Bentzen Winje, Lukas Fehring, Marius Lindauer

llms

arXiv AI

Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento

llms benchmarks

One email a morning, machine-written

Related stories

TANGO: Treating Tokens as Operators

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training