arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv:2610.32104v1 Announce Type: cross Abstract: Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing differen...
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...
arXiv:2606. 19697v1 Announce Type: cross Abstract: The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by theoretical results showing that chain-of-thought (CoT) transformers can simulate Turing machines, and thus perform arbitrary computation.
arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.
The paper presents a method for synthesizing and verifying Deterministic Register Automata (DRAs) over ordered data sequences such as time series and sensor streams. It introduces a local‑search synthesis procedure and an exact robustness checker that runs in polynomial time for a fixed number of registers, reducing robustness to a shortest‑path problem. The framework, implemented in RAExc, is applied to learn interpretable surrogate models for neural sequence classifiers, providing statistical guarantees of fidelity and local stability while verifying robustness on benchmark languages and sequentialized MNIST.
arXiv:2603. 17019v2 Announce Type: replace Abstract: A central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples.
arXiv:2604.13082v3 Announce Type: replace-cross Abstract: Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers...
A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque.
M$^2$RNN introduces matrix‑valued hidden states and expressive non‑linear transitions for recurrent neural networks, addressing the limited expressive power of transformers. The architecture scales efficiently with tensor cores, achieving perfect state‑tracking generalization on unseen sequence lengths and outperforming hybrid Gated DeltaNet models by 0.4–0.5 perplexity points on a 7B MoE model while using three‑times smaller recurrent states. Even a single M$^2$RNN layer in a hybrid architecture yields accuracy gains comparable to full Hybrid M$^2$RNN, and improves long‑context performance on LongBench by up to 8 points over state‑of‑the‑art hybrid linear attention models.
arXiv:2607. 04319v1 Announce Type: cross Abstract: A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality.