arXiv:2608. 14691v1 Announce Type: new Abstract: Sequence models are conventionally distinguished by their backbone, the mechanism that routes information across positions, such as attention or recurrence.
By Ahmed Nebli, Hadi Saadatdoorabi, Christopher Keibel, Kevin Yam
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.
By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
arXiv:2610.16301v1 Announce Type: new
Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)
arXiv:2608.21308v1 Announce Type: new
Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the ex...
By Zeyun Zhong, Joya Chen, Manuel Martin, Frederik Diederichs, Juergen Gall, Juergen Beyerer
TraceRelay is an attention‑aligned recurrent architecture that distributes persistent representations over a rolling sequence of low‑dimensional traces. It uses right‑looking attention to form increments from lower‑layer representations, delays delivery until all attended inputs are in the causal past, and accumulates these increments with a fixed additive phase recurrence. Experiments on 36 small‑model runs show that recurrent phase inheritance yields high accuracy on Equal Repeats (up to 99.69%) but accuracy drops beyond the training range, and that the optimal size of recurrent traces varies across tasks such as Dyck closing‑type prediction and causal Most‑Freq generation.
By Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
TraceRelay is an attention‑aligned recurrent architecture that spreads persistent representations across a rolling sequence of low‑dimensional traces. It uses right‑looking attention to build increments, delays their delivery until all attended inputs are in the causal past, and then applies a fixed additive phase recurrence to accumulate them; left‑looking attention reads the resulting residual‑augmented stream. Experiments on 36 small models across tasks such as Equal Repeats, bounded Dyck closing‑type prediction, and causal Most‑Freq generation show that models with recurrent phase inheritance achieve high accuracy within the trained length, while performance drops sharply beyond that range, and that the optimal size of recurrent traces varies by task.