arXiv AI
Aug 11
Full-bandwidth transformer
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
The Von-Neumann State-Space Transformer for neural decoding
The paper introduces the Von‑Neumann State‑Space Transformer (VN‑SST), a memory‑augmented Transformer that replaces the standard feed‑forward block with a low‑rank instruction bank. By decoding token‑specific operators from a low‑dimensional state‑space memory, VN‑SST achieves higher data‑efficiency and parameter‑efficiency on motor‑cortex neural‑decoding tasks and on small language‑model benchmarks. The model demonstrates that a compact instruction set can act as a control channel, improving performance without increasing accuracy through larger parameter counts.
Related stories
arXiv Computation and Language
4d ago
Pretraining Latent Information Feedback Transformers with Teacher Supervision
arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...
arXiv AI
Aug 18
Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation
arXiv:2608. 12385v2 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training.
arXiv Machine Learning
Sep 21
Trading Depth for Time in Recurrent Transformers
The paper investigates whether extra computation in recurrent Transformers should be allocated to more temporal steps or greater physical depth. Using Latent Recurrent Transformers (LRTs), the authors insert a latent thought token between vocabulary tokens, allowing each token to pass through the same $L$ layers twice while sharing parameters. Experiments on 16‑ and 20‑layer mixture‑of‑experts NanoChat backbones show that a single thought token brings a shallower model within 0.006–0.004 bits per byte of a double‑depth counterpart, recovering 67–81% of the improvement with roughly 48% fewer parameters.
Hugging Face Trending Papers
Jul 9
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.
arXiv Machine Learning
Sep 2
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
arXiv:2605.26797v2 Announce Type: replace Abstract: We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden...
arXiv Machine Learning
Jun 8
Sparsely gated tiny linear experts
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
arXiv AI
Jun 29
The Context-Ready Transformer
arXiv:2606. 27538v1 Announce Type: cross Abstract: We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block that pre-contextualizes each token before it enters the block.
arXiv AI
Jul 9
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
arXiv:2607. 06601v1 Announce Type: cross Abstract: Conditional computation can decouple language model quality from per-token inference cost, yet leading techniques act on a single axis in isolation: Mixture-of-Experts (MoE) sparsifies the FFN, Mixture-of-Depths (MoD) skips whole transformer blocks, and KV-cache quantization compresses attention memory.
arXiv AI
Aug 14
Dual-Flow Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation
arXiv:2608. 12385v1 Announce Type: new Abstract: As large language models serve more requests, cumulative inference cost is becoming increasingly important relative to one-time training cost.
arXiv Computation and Language
Sep 24
Distilling Sequential Computation in Transformer Language Models
arXiv:2609.27233v1 Announce Type: new Abstract: Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adja...
arXiv Machine Learning
Jun 17
Olmo Hybrid: From Theory to Practice and Back
arXiv:2604. 03444v4 Announce Type: replace Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention.