The paper introduces an input‑dependent readout phase for ResonatorLM, a resonant sequence model that traditionally uses a fixed phase per head to read memory. By making the phase a function of the current token, the authors add input dependence to the read‑only operation while preserving the exact convolutional path and adding only 0.2 % more parameters. Experiments on 6 M‑parameter character‑level models show that this modification matches the original performance at the training length and dramatically improves generalisation to longer sequences, reducing perplexity from 12.81 to 3.76 when trained on 256 tokens and tested on 1024 tokens.
arXiv:2608. 14691v1 Announce Type: new Abstract: Sequence models are conventionally distinguished by their backbone, the mechanism that routes information across positions, such as attention or recurrence.
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
arXiv:2610.16301v1 Announce Type: new Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...
TraceRelay is an attention‑aligned recurrent architecture that distributes persistent representations over a rolling sequence of low‑dimensional traces. It uses right‑looking attention to form increments from lower‑layer representations, delays delivery until all attended inputs are in the causal past, and accumulates these increments with a fixed additive phase recurrence. Experiments on 36 small‑model runs show that recurrent phase inheritance yields high accuracy on Equal Repeats (up to 99.69%) but accuracy drops beyond the training range, and that the optimal size of recurrent traces varies across tasks such as Dyck closing‑type prediction and causal Most‑Freq generation.
TraceRelay is an attention‑aligned recurrent architecture that spreads persistent representations across a rolling sequence of low‑dimensional traces. It uses right‑looking attention to build increments, delays their delivery until all attended inputs are in the causal past, and then applies a fixed additive phase recurrence to accumulate them; left‑looking attention reads the resulting residual‑augmented stream. Experiments on 36 small models across tasks such as Equal Repeats, bounded Dyck closing‑type prediction, and causal Most‑Freq generation show that models with recurrent phase inheritance achieve high accuracy within the trained length, while performance drops sharply beyond that range, and that the optimal size of recurrent traces varies by task.
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
As new evidence arrives, a sequence model must update what it remembers and how memory influences predictions. While Transformers incur computation and cache costs scaling with context length, fixed-s...
Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement.
arXiv:2606. 31495v1 Announce Type: new Abstract: We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition.
arXiv:2609.39082v1 Announce Type: new Abstract: As new evidence arrives, a sequence model must update what it remembers and how memory influences predictions. While Transformers incur computation and...
arXiv:2607. 02303v1 Announce Type: new Abstract: Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades.