arXiv AI
Parity, Sensitivity, and Transformers
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 1
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and lengt...
arXiv Machine Learning
Jul 21
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
arXiv:2607. 17624v1 Announce Type: new Abstract: Transformers are remarkably versatile and their design is largely consistent across a variety of applications.
arXiv Machine Learning
Jul 7
Legible-by-Construction: Attention and End-to-End Transformers
arXiv:2607. 04319v1 Announce Type: cross Abstract: A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality.
Hugging Face Trending Papers
Jul 5
Legible-by-Construction: Attention and End-to-End Transformers
A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque.
arXiv Machine Learning
Jul 27
Indexing: the Beginning and the End
arXiv:2607. 22361v1 Announce Type: new Abstract: We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive.
arXiv Machine Learning
Sep 2
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
The paper examines how to allocate attention heads and head dimensions across Transformer layers to balance expressivity and efficiency. It provides a mathematical analysis of early layers’ role in information extraction and characterizes the trade‑off between head count and dimension under a fixed parameter budget. The authors prove a saturation effect of softmax activations, showing that increasing head dimensions yields diminishing returns, especially for long sequences, and propose strategies for efficient parameter allocation across layers.