arXiv:2606. 07254v1 Announce Type: new Abstract: State tracking exposes a sharp limitation of sequence models: the relevant signal is often not a summary of observed tokens, but an ordered latent state that evolves through non-commutative transformations.
By Jeonghoon Lee
arXiv:2603.19198v3 Announce Type: replace
Abstract: We introduce the exponentially weighted signature (EWS), a continuous-time model that computes iterated integrals of a path, where each increment i...
By Alexandre Bloch, Benjamin Walker, Jo\"el Mouterde, Sam Morley, Samuel N. Cohen, Terry Lyons
arXiv:2608. 03425v1 Announce Type: new Abstract: Transformer-based architectures have dominated sequence modeling, largely due to the expressive power of attention mechanisms.
By Xiaohe Li, Yang Lu
arXiv:2602. 10743v2 Announce Type: replace Abstract: State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but their linear state updates limit expressivity and robust state tracking.
By Vaisakh Shaj, Cameron Barker, Aidan Scannell, Andras Szecsenyi, Elliot J. Crowley, Amos Storkey
The paper investigates how neural networks, particularly Transformers and recurrent models, learn to track group elements by predicting their running product. It finds that Transformers tend to recover quotient classes and that their accuracy can be predicted by the reciprocal of class size, while recurrent networks can capture both normal and non‑normal right‑coset partitions. The study links partial accuracy, learning stages, and internal state representations to the subgroup cosets the models learn to track.
By Zhiyu Zhang, Yupeng Li
arXiv:2604. 14501v2 Announce Type: replace-cross Abstract: We study how depth, finite precision, state dimension, and chain-of-thought (CoT) affect the expressive power of multi-layer state-space models (SSMs).
By Nikola Zubi\'c, Qian Li, Yuyi Wang, Davide Scaramuzza